acceptodds
Under review as a conference paper at ICLR 2027

Quantization Beyond Matrix Multiplication for Efficient LLM Inference

Abstract

Quantization is widely used to accelerate LLMs inference, but existing approaches primarily optimize matrix multiplication while leaving surrounding operators and data movement in high precision. This limits end-to-end benefits of quantization, particularly for memory-bound operators. In this paper, we proposed an INT8 pipeline that extends quantization beyond matrix multiplication. Our design preserves intermediate tensors in INT8 across both matmul and non-matmul operators, reducing global-memory traffic. We further introduce a hierarchical warp–block reduction strategy that effectively utilize on-chip resources and maintains high GPU occupancy during compute deep learning operators. We evaluate our approach on Qwen3 and Llama-3.2 models across multiple tasks and workloads. Compared with BF16 inference, our approach achieves up to inference speedup and reduces energy consumption by up to 39%, while maintaining comparable accuracy. The benefits remain consistent across GPUs, decoding workloads, and batch sizes, demonstrating that quantization beyond matmul is important for efficient end-to-end LLM inference.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
LLM Quantization, GPU Kernel Optimization

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.20 over 5 reviews, above 49% of the others.
Reviewer_poQTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper develops an INT8 LLM inference pipeline that stores intermediate tensors in INT8 across supported matmul and non-matmul operators, while performing sensitive computations in floating point on chip. It combines fused output quantization with reduction kernels that avoid full-vector storage through recomputation. Experiments on Qwen3 and Llama-3.2 report latency and energy improvements with modest quality degradation, including gains alongside FlashAttention and torch.compile.

Critical strengths
  1. The paper presents a coherent implementation of low-precision dataflow across supported operator boundaries. It keeps tensor storage in INT8 while retaining floating-point computation on chip, and fuses matmul output requantization to avoid higher-precision global-memory intermediates. For non-matmul operators, it accounts for the additional reductions introduced by output-scale computation, even for element-wise functions, and uses recomputation to limit temporary storage. These design choices address both global-memory traffic and on-chip resource constraints in the inference pipeline.
  2. The evaluation examines model quality together with latency and energy, and checks the pipeline across different models, GPUs, batch sizes, and autoregressive decoding workloads. Of particular value is the optimized configuration: both BF16 and INT8 use FlashAttention and torch.compile, and the evaluated INT8 pipeline retains latency and energy benefits. This provides evidence that the complete implementation has practical value alongside established inference optimizations, strengthening the case for the work as a systems contribution.
Critical weaknesses
  1. The resource-efficient kernel contribution is not sufficiently distinguished from existing implementations. Section 4.1 and Appendix A.2 clearly describe the reduction and recomputation strategy, and Tables 7–8 substantiate its performance and resource benefits against the reported baselines. However, an existing vLLM v0.8.5 RMSNorm–quantization kernel already uses the same broad three-stage organization with block-reduction temporary storage instead of a full-row buffer. Its floating-point inputs differ from the scaled INT8 inputs of Algorithm 1. The paper does not clearly distinguish which aspects of the resource strategy constitute a technical advance over this existing organization and which are adaptations to the INT8 interface. This limits the novelty assessment of a stated main contribution, while leaving the demonstrated implementation benefits intact.
  2. The additional end-to-end benefit beyond INT8 matmul remains insufficiently isolated in the optimized setting. Table 2 provides comparisons with existing quantization methods, Figure 4 demonstrates reduced non-matmul execution time, and Table 6 shows useful gains alongside FlashAttention and torch.compile. However, these results do not isolate the incremental gain from the extended non-matmul dataflow over a matched matmul-only INT8 pipeline under the optimized configuration. Table 6 changes both matmul precision and the surrounding dataflow relative to BF16, leaving the magnitude of this additional benefit unresolved.

I lean toward weak rejection because the technical advance over existing kernel implementations is insufficiently established, and the optimized experiments do not isolate the additional end-to-end benefit of extending quantization beyond matmul. The demonstrated practical benefits are meaningful, and clarification of these two points could change my assessment.

Questions for authors
  1. For RMSNorm, which aspects of Section 4 constitute new kernel techniques, and which adapt existing fused normalization–quantization designs? Please clarify the incremental contribution relative to the vLLM implementation cited above, accounting for the different input formats.
  2. Under the FlashAttention and torch.compile configuration of Table 6, what additional end-to-end gain comes from extending INT8 storage through the surrounding non-matmul operators beyond using INT8 matmuls? A representative paired comparison could match matmul backends and optimization settings, while allowing necessary output-format and epilogue differences.
2 comments

We thank the reviewer for these valuable questions and suggestions.

Q1. Incremental contribution relative to vLLM

We acknowledge that vLLM already implements an efficient fused RMSNorm–quantization kernel using a three-stage computation and block-level reductions. Our work builds on these established GPU techniques rather than introducing a fundamentally new reduction algorithm. The key distinction lies in the input/output precision and integration into the inference pipeline. The cited vLLM kernel accepts floating-point inputs and produces quantized outputs, whereas our RMSNorm kernel directly consumes INT8 inputs, performs higher-precision computation on chip, and writes INT8 outputs. Furthermore, our contribution extends beyond RMSNorm by integrating INT8 GEMM, GEMV, Softmax, and SiLU into a low-precision dataflow that preserves INT8 representations across supported operators. We will clarify this distinction in the revised manuscript.

Q2. Incremental benefits beyond INT8 matmul.

Section 5.4 (Table 6) compares SmoothQuant (INT8 matmul-only) with our full INT8 pipeline under the same FlashAttention and torch.compile optimization settings. Across five models, our pipeline provides an additional 1.03–1.07× prefill speedup and approximately 3–9% energy savings over INT8 matmul-only execution. While these gains are relatively small, they demonstrate that preserving INT8 intermediate representations remains beneficial under optimized execution. The modest improvements may partly reflect our reliance on custom CUDA kernels, which are not automatically fused with surrounding operators. We acknowledge this limitation and identify tighter integration with graph compilers and operator fusion as important directions for future work.

Thank you for the response. My score remains unchanged.

Reviewer_MJzpOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper argues that matmul-centric quantization leaves most of the inference pipeline in BF16, so non-matmul operators and intermediate traffic dominate cost. It proposes a W8A8O8 pipeline that keeps tensors in INT8 in global memory across GEMM/GEMV and non-matmul operators (RMSNorm, Softmax, SiLU), dequantizes on chip, computes in higher precision, and requantizes before writeback. A hierarchical warp–block reduction bounds shared-memory use for reduction operators. On Qwen3 / Llama-3.2 across several GPUs, the pipeline reports up to 1.6× speedup and 39% energy reduction vs BF16, with only modest PPL/zero-shot degradation.

Critical strengths

Clear systems framing: non-matmul ops are memory-bound and a large share of prefill latency; treating activation quantization as a data-movement problem is the right lens. Hierarchical warp–block reduction is a practical fix for shared-memory pressure and occupancy (naive INT8 can be slower than BF16). Evaluation is fairly broad for a systems paper: multiple models, GPUs, batch sizes, decoding, and a FlashAttention + torch.compile setting.

Critical weaknesses

Novelty is thin for ICLR. The hierarchical reduction is standard CUDA practice (warp shuffle + one shared slot per warp; Algorithms 2–3 are textbook). The INT8-in/INT8-out dataflow is close to Jetfire / I-BERT / IntAttention; the paper does not isolate an algorithmic ML contribution beyond kernel engineering. Quality loss is understated. PPL and zero-shot drop are non-trivial (e.g. Llama-3.2-1B PPL 11.99→12.56; Qwen3-4B ARC-C 70→68; Llama-3.2-1B ARC-C 37→35), with no generation quality or long-context eval. Residual norms are kept in BF16, so “quantization beyond matmul” is only partial. Headline speedup is confounded. 1.6× is mainly vs PyTorch-eager BF16; under FlashAttention + torch.compile gains fall to 1.1–1.5× (Table 6). No comparison to modern serving stacks (vLLM / TensorRT-LLM / Jetfire) or compiler-optimized BF16 as the primary baseline; SmoothQuant/QuaRot latency comparisons are not clearly matched on fusion/compiler settings. Calibrated output scales (s_Y in fused GEMM) lack robustness analysis. Appendix D shows separate vs fused differ in RMSE, but there is no study of scale drift under distribution/length shift — a risk for production INT8 pipelines.

Questions for authors

How stable are calibrated s_Y values under longer contexts or domain shift, and what happens to accuracy if they drift? Please compare against torch.compile-only BF16 and an existing INT8 serving path (Jetfire / TensorRT-LLM / vLLM), not only eager BF16 and SmoothQuant/QuaRot. Can you report long-context and generation quality (e.g. longer PPL, summarization/translation), not only ARC/PIQA? Why are residual norms left in BF16 — does a full-INT8 version fail accuracy, and what is the measured cost of keeping them?

1 comment

We thank the reviewer for these valuable suggestions.

  1. Calibration stability. We evaluate our INT8 pipeline at longer context lengths of 4,096 and 8,192 tokens, along with long-form text generation. These experiments show that quantization errors do not substantially increase within the evaluated context lengths. However, they do not directly measure the stability of output scales (s_Y). A systematic study of domain shift remains outside the scope of this work and is an important direction for future research.

  2. Optimized inference baselines. Section 5.4 compares our method against BF16 and SmoothQuant under three configurations: FlashAttention only, torch.compile only, and their combination. These results demonstrate that our approach remains beneficial beyond eager execution. We acknowledge the importance of comparisons with existing LLM serving frameworks such as TensorRT-LLM and vLLM. However, these frameworks incorporate additional serving-level optimizations, including scheduling, batching, and KV-cache management, which introduce backend differences beyond our operator-level quantization design. We therefore leave integration with these frameworks for future work.

  3. Long-context and generation quality. We extend our evaluation in Appendix G to include perplexity at context lengths of 4,096 and 8,192 tokens. We also provide qualitative generation examples covering translation, information extraction, question answering, and long text generation.

Reviewer_DfGaOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper maintains INT8 representations across supported matmul and non-matmul boundaries, while dequantizing and carrying out sensitive computations on chip. It combines fused GEMM requantization, a GEMV implementation, and hierarchical reductions that avoid materializing full floating-point vectors in shared memory. Tests cover five Qwen/Llama models, multiple GPUs, decoding and prefill, and an optimized setting using FlashAttention and torch.compile. The key claim is that reducing intermediate memory traffic provides benefits beyond low-precision matmul alone.

Critical strengths
  1. Sections 3–4 explain how INT8 input/output with higher-precision on-chip computation can reduce materialized traffic. Tables 7–8 show that a naive INT8 implementation can perform poorly, while a resource-conscious design improves occupancy and latency. This is useful implementation evidence.
  2. Table 6 uses FlashAttention and torch.compile for both precisions and still reports latency gains, ranging from approximately 1.1× to 1.5×. This materially strengthens the paper beyond the eager-mode profiling in Figure 2; it would be incorrect to describe the entire evaluation as an eager-only comparison.
Critical weaknesses
  1. The end-to-end results do not isolate the incremental contribution of non-matmul quantization.The proposed pipeline changes matmul kernels, output epilogues, intermediate formats, and non-matmul kernels together. SmoothQuant and QuaRot comparisons in Table 2 are not an otherwise-identical matmul-only version of this implementation. Table 6 addresses compiler/fusion compatibility, Tables 7–8 isolate reduction designs, and Appendix D tests a GEMM epilogue, but these still leave the central end-to-end attribution unresolved. A matched backend with non-matmul INT8 boundaries toggled on/off is needed to establish how much benefit specifically comes from going beyond matmul (pp. 7–9; Appendix D).
  2. The precision and workload specifications leave important ambiguity. Section 3 retains normalization between attention and feed-forward modules in BF16, while Section 5.4 lists RMSNorm among the remaining INT8 operators in the optimized pipeline. The GEMM output scale is calibrated, but the PDF does not clearly specify the full calibration corpus, sample count, or mapping of static scales to variable sequence positions. Table 4 labels its size column as generated length, while the surrounding discussion explains TTFT changes in terms of prompt length; both lengths need reporting. These omissions affect replication and interpretation of the quality/latency trade-off, especially for repeatedly requantized softmax or activation outputs (pp. 3–4, 6–8).
  3. The headline "up to 1.6× inference speedup" is an eager-mode prefill (TTFT) number; complete generation improves by only about 1.1×. The latency column of Table 2 is identical, model by model, to the batch-1 TTFT column of Table 3 (e.g., 986 → 620 ms for Qwen3-14B), and Table 3's batch-4 and batch-16 TTFT speedups are similar (1.4–1.6×). The 37% reduction (L. 303) and the 1.6× headline (986/620 ≈ 1.59; L. 20–21, L. 95, L. 484–485) therefore measure prefill only, across batch sizes 1–16. The 39% energy figure likewise comes from these short runs (Table 2: 36 → 22 J for Llama-3.2-1B; Table 6: 43 → 26 J for Llama-3.2-3B), and no energy is reported for decoding. For complete generation, Table 4 (Qwen3-14B) gives total-time ratios of 1.12×, 1.14×, and 1.11× (7.6 → 6.8 s, 32 → 28 s, 68 → 61 s) and ITL ratios of 1.11–1.14×, while INT8 TTFT at the shortest length is 60% higher (78 → 125 ms). With FlashAttention and torch.compile, the Qwen3 gains are 1.1–1.3× (Table 6). The abstract nevertheless says that the benefits "remain consistent across different GPUs, decoding workloads, and batch sizes" (L. 22–23; similarly L. 484–485), and the paper has no limitations section. The abstract and conclusion should identify the 1.6×/39% figures as prefill (TTFT) measurements and report end-to-end request latency for representative prompt/output lengths.
  4. The cross-GPU results show that the non-matmul benefit depends strongly on the hardware and is smallest on the HBM-equipped A100 and A30. In Fig. 4 (p. 7), total time falls 1.76× on the RTX 5000 Ada (439 → 249 ms) and 1.52× on the A40 (647 → 426 ms), but only 1.32× on the A30 (583 → 441 ms) and 1.14× on the A100 (306 → 269 ms). The non-matmul segment, which carries the paper's mechanism, shrinks by 55% and 46% on the two GDDR6 GPUs but by only 22% (302 → 234 ms) and 10% (142 → 128 ms) on the A30 and A100. INT8 does reduce both components on every GPU, as stated at L. 313–314, but the size of the benefit is far from architecture-independent, and the paper does not discuss why. One possibility is that on high-bandwidth GPUs the extra passes and reductions of the INT8 kernels, rather than DRAM traffic, become the bottleneck; per-kernel achieved bandwidth on each GPU would clarify this. This matters because datacenter GPUs of the A100 class and newer are arguably the main deployment target. The caption (L. 359) names neither the model nor the workload (the A40 totals closely match the Qwen3-8B prefill of Fig. 2 and Table 2), and the y-axis reports "CUDA Time", presumably summed kernel time, rather than the wall-clock latency used elsewhere.
  5. During decoding, the reported gains are attributable to INT8 weights in the GEMV rather than to quantization beyond matmul, and weight-only baselines are missing. By the paper's own Table 1(b) (p. 3), the listed non-matmul operators move almost no data in a decode step: 0.3M for softmax and 0.06M for SiLU, against 50–170M for each projection, i.e., under 0.1% of the listed decode traffic (0.36M of 408.36M). Halving that traffic cannot measurably change inter-token latency. The paper's explanation of the decoding gain (L. 319–321) instead rests on the INT8 GEMV (Table 5), whose benefit comes from 1-byte weights; weight-only W8A16 or W4A16 schemes such as GPTQ and AWQ (cited at L. 45–46 but not compared) provide this without quantizing activations. Moreover, ITL improves by only 10–12% (Table 4) despite the ≈2.4× faster GEMV (see Question 4). Decoding is evaluated for a single model and three lengths, at a batch size that is not stated (the throughput column equals 1000/ITL, consistent with batch 1), and without continuous batching. The "decoding workloads" claim (L. 22–23) therefore does not yet support quantization beyond matmul for the decode phase.

Others:

  • Fig. 1 (p. 1): panel (c) colors "Normalization" green (INT8), whereas Sec. 3 (L. 152–154) and Sec. 5.1 (L. 286–287) keep the normalization layers between modules in BF16. The figure colors operators, although the caption speaks of tensor representations, and its red/green-only encoding is not colorblind-safe.
  • Fig. 4 (L. 359): besides the missing model and workload (Weakness 4), the legend says "Bf16"/"Int8", while the axis labels and text say "BF16"/"INT8". The in-bar values are very small.
  • Fig. 2 (L. 143) and Fig. 5 (L. 927): axis labels and in-bar text are too small to read at print size. The "Other" segment of Fig. 2 carries no value, and its percentages appear truncated rather than rounded (e.g., for Qwen3-14B, 500/970 = 51.5% is shown as 51% and 310/970 = 32.0% as 31%). Fig. 5(c) has no legend and no RMSE units, and "GEMM Dimension" (M = N = K?) is not defined.
  • The same BF16 configuration has slightly different totals in different places: 970 ms (Fig. 2) vs. 986 ms (Table 2) for Qwen3-14B, and 641 ms (Fig. 2), 647 ms (Fig. 4, A40, if this is Qwen3-8B), and 651 ms (Table 2) for Qwen3-8B. These are presumably separate runs, so please report repetitions and variability (cf. Question 3).
  • Table 1 (L. 108): the caption says "Profiling", but every entry matches an analytical count. Rows that combine two projections report per-projection values: "q proj & o proj" = 107G = 2 × 2048 × 5120 × 5120 is the cost of a single 5120 × 5120 projection. "Memory Traffic" is in MiB of BF16 bytes (e.g., 90 MiB), while FLOPs use decimal G. Please state that values are per layer and per projection, and give the units. Table 1 and Table 9 also report different BF16 SiLU arithmetic intensities (≈1 vs. 0.6–0.7) without explaining the counting.
  • Tables 3–5: Table 3 mixes ms and s within one table. Table 4's "Thr." column is simply 1000/ITL, and three of its six entries are truncated rather than rounded (1000/64 = 15.6 shown as 15, 1000/67 = 14.9 as 14, 1000/60 = 16.7 as 16). Table 5's GEMV shapes (4096–16384) do not match the Qwen3-14B projection shapes used in Table 4 (5120 × 5120, 5120 × 1024, 5120 × 17408), and its "M, N" labels conflict with Sec. 3.1, where M indexes tokens and K is the reduction dimension.
  • Reporting precision: Table 6 gives energy in whole joules for values as small as 7–11 J, so Llama-3.2-1B's "36%" saving is compatible with ≈29–43% given rounding. Table 7's two-decimal latencies (0.07 ms) cannot resolve differences at d = 4096.
  • L. 413 ("allows up to 11 blocks per SM"): 11 is the shared-memory-only limit defined at L. 408–410. With B = 256 (L. 720), the A40's limit of 1,536 resident threads per SM caps residency at 6 blocks. The reported occupancies are consistent with this: 90–92% achieved against 100% theoretical (6 × 8 = 48 warps) for the hierarchical kernels, and 78–80% and 33% for the naive kernels (5 and 2 blocks of 8 warps, i.e., 83% and 33% theoretical).
  • Algorithms: Algorithm 4 jumps from "Step 2" to "Step 4" (L. 776). Algorithm 5(b) sets s_y ← M/127 without the ε_q floor used in Algorithms 1 and 4 (L. 798 vs. L. 235), and ε_q is never defined. In Algorithm 3, the first warp reduces smem[ℓ] across all 32 lanes (L. 674), although only N_w = B/W entries are written; the remaining entries should be set to OP's identity. BROADCAST (Algorithms 1 and 4) is not defined.
  • Notation is overloaded. R denotes the number of rows (L. 210), the sequence length (L. 707), the sum of squares (Algorithm 1, App. C.2), and the softmax denominator (Algorithm 4). M is both a reduction result (Algorithms 1, 4, and 5) and a matrix dimension (Table 5). W is the warp size, the work counts in App. C, and part of the weights W_q. d is called the hidden dimension (L. 707, L. 744–745), although for attention softmax the row runs over key positions.
  • Undefined acronyms and labels: TTFT, ITL, SM, SIMT, NCU, SMEM, AI (Table 9), GEMM/GEMV, and "W8A8O8" (L. 301) are used without definition.
  • Inconsistent naming: "Float"/"BF16" (Table 2); "Ours"/"W8A8O8"/"INT8" for the same method; model names in three styles ("Qwen3-4B" in Tables 2 and 6, "Qwen3 4B" in Fig. 2, "Qwen3–4B" with an en dash in Table 3, and likewise for Llama-3.2); and inconsistent "softmax"/"Softmax" capitalization.
  • Some numerical statements do not match the paper's own data. "matmul accounts for only 30%–50%" (L. 58; "approximately 30%–50%" at L. 142) and "non-matmul operators account for 50%–70%" (L. 84) conflict with Qwen3-14B in Fig. 2, which is 51.5% matmul and ≈48.5% non-matmul. The same SiLU intensity ratio is "more than 2.6×" at L. 275 but "more than 2.7×" at L. 809; Proposition 1 (L. 835) establishes only 8/3 ≈ 2.67, so the L. 809 bound does not follow for all C_σ. L. 949–951 claims consistency with the "lower memory utilization" in Table 9, but Table 9 profiles SiLU only, and at d = 16384 its INT8 memory utilization (85%) is close to BF16's (88%).
Questions for authors
  1. Under the same FlashAttention/compiler/backend configuration, what are latency, energy, and quality for BF16, your matmul-only INT8 pipeline, and the full INT8-boundary pipeline? Which operator boundaries account for the incremental gain?
  2. Please provide an operator-by-operator precision/scale map for both eager and optimized execution, including which RMSNorm operations remain BF16, and the calibration data/scale assignment for GEMM outputs.
  3. Please specify prompt and output lengths, repetition/warm-up protocol, timing synchronization, and energy measurement procedure for the headline comparisons. For quality, can you show uncertainty and at least one longer-form generation evaluation to test cumulative non-matmul requantization effects?
  4. Decoding: Could you provide a per-operator breakdown of the Qwen3-14B decode step for BF16 and INT8, and a comparison with a weight-only INT8 (W8A16) GEMV, and ideally W4A16, in the same backend? As a rough bound, I used the layer dimensions implied by Table 1 together with the public configuration (40 layers and an untied 151,936 × 5,120 output head). Streaming the weights at the A40's 696 GB/s then takes ≈40 ms per token in BF16 and ≈20–21 ms in INT8, whereas the measured ITL is 61 vs. 55 ms (Table 4). This implies ≈21 ms of non-weight time per token for BF16 but ≈34–35 ms for INT8, even though Table 5 reports a ≈2.4× faster INT8 GEMV. What accounts for the extra INT8 time? If the decode gain comes only from INT8 weights, the benefit of going beyond matmul would be confined to prefill.
  5. Optimized baseline: For the Table 6 configuration, please state the workload and provide an operator-level breakdown like Fig. 2/Fig. 4 for both BF16 and INT8. What fraction of compiled-BF16 prefill latency remains in non-matmul operators after FlashAttention and torch.compile? If that share is small (Weakness 6), the motivation and the Analysis contribution would need to be restated; Question 1 covers the corresponding attribution of the INT8 gain.
  6. BF16 references: What exactly are the BF16 implementations in Tables 5, 7, and 9 and in the eager end-to-end model (cuBLAS, PyTorch/Hugging Face composite operators, or custom kernels)? Could you add BF16-in/BF16-out versions of your custom kernels with the same passes and fusion, and report achieved DRAM bytes and bandwidth (e.g., from Nsight Compute) for each BF16/INT8 pair? Given speedups that exceed the 2× byte ratio (Weakness 7), this is needed to separate the precision effect from the implementation effect.
  7. State-of-practice comparison: At matched batch size and sequence lengths, how does the pipeline compare with a production W8A8 INT8 path (e.g., TensorRT-LLM or vLLM) and, on the RTX 5000 Ada, with FP8 storage and compute? If those stacks already fuse quantization into normalization and GEMM epilogues, what additional traffic does your design remove?
  8. INT8 softmax and long context: What fraction of the attention probability mass is rounded to zero by the per-row 7-bit softmax quantization of Algorithm 4, by layer and head, at 2K–8K contexts? How do BF16 and INT8 compare on perplexity at ≥ 8K tokens and on a long-context retrieval task? Please also report quality for the optimized configuration of Table 6, which differs numerically from the eager pipeline evaluated in Table 2.
1 comment

We thank the reviewer for the detailed and constructive feedback. We appreciate the recognition of our motivation, kernel design, and optimized inference results.

Q1. Controlled comparison and incremental benefits

Section 5.4 (Table 6) compares BF16, SmoothQuant (INT8 matmul-only), and our full INT8 pipeline under the same FlashAttention and torch.compile optimization settings. Across five models, our pipeline provides 1.03–1.07× prefill speedup and approximately 3–9% energy savings over INT8 matmul-only execution. Although these improvements are modest, they demonstrate that preserving INT8 representations remains beneficial under optimized execution. Our custom CUDA kernels are also not automatically fused with surrounding operators, leaving opportunities for further compiler integration.

Q2. Numerical precision and calibration

We have added an operator-level precision-flow diagram in Appendix C (Figure 5) to clarify tensor representations throughout the Transformer. Our pipeline preserves INT8 inputs and outputs across supported operators. Residual connections and boundary normalization layers remain in BF16 for numerical stability. GEMM output scales are calibration-derived using 10,000 calibration samples, whereas non-matmul kernels dynamically determine output scales through row-wise reductions.

Q3. Experimental methodology and generation quality

Section 5.1 specifies 10,000 executions for individual operators and 100 runs for model inference. GPU energy is estimated using NVML power measurements and execution time. Table 4 evaluates Qwen3-14B using matching prompt and generation lengths of 128, 512, and 1,024 tokens. We have also clarified the distinction between prefill and autoregressive generation: our method achieves up to 1.6× prefill speedup, while total generation time improves by 1.14×. Additionally, Appendix G provides perplexity evaluations at 4,096 and 8,192 tokens and qualitative generation examples covering translation, information extraction, question answering, and long-form generation.

Q4. Decoding performance

We agree that decoding performance is strongly influenced by weight movement and GEMV efficiency. Table 5 demonstrates approximately 2.4× lower latency for our INT8 GEMV kernels, while Table 4 reports smaller end-to-end improvements. We agree that a matched W8A16/W4A16 comparison and detailed per-operator decoding breakdown would help isolate the contribution of non-matmul quantization during decoding. We acknowledge this limitation and leave a more detailed decoding analysis for future work.

Q5. Optimized inference breakdown

We agree that the operator distribution changes under FlashAttention and torch.compile. Figure 2 characterizes eager execution, while Section 5.4 evaluates optimized inference separately. Table 6 shows that the incremental benefits beyond INT8 matmul decrease to 1.03–1.07× under combined optimization. This suggests that compiler fusion may overlap with some of the benefits of our custom kernels. We acknowledge that an additional operator-level breakdown under optimized execution would help explain this reduction and leave a more detailed analysis for future work.

Q6. BF16 baselines and kernel implementation

We agree that matched BF16 and INT8 kernel implementations would help distinguish precision-related improvements from implementation optimizations. Our current evaluation compares BF16 execution with our optimized INT8 kernels, while Section 5.5 (Tables 7–8) isolates the impact of hierarchical reduction by comparing naive and hierarchical INT8 designs under the same precision. Appendix F (Table 11) further provides GPU profiling results explaining the improvements in hardware utilization. We acknowledge that a controlled comparison using matched BF16-in/BF16-out kernels and direct DRAM-traffic measurements would provide additional insight. We leave this more detailed investigation for future work.

Q7. Comparison with serving frameworks

We acknowledge the importance of comparisons with production inference frameworks such as vLLM and TensorRT-LLM. Our work focuses on operator-level quantization and maintaining INT8 representations across supported operators. We do not claim superiority over serving systems, which incorporate optimizations such as scheduling, batching, and KV-cache management. Integration with these frameworks and extensions to FP8 remain important directions for future work.

Q8. INT8 Softmax and long-context accuracy

We have extended our evaluation in Appendix G (Table 12) to include perplexity at context lengths of 4,096 and 8,192 tokens. The results show that perplexity degradation remains relatively stable within the evaluated context lengths. We additionally provide qualitative examples of translation, information extraction, question answering, and long-form text generation. A detailed analysis of attention probability mass removed by INT8 quantization remains a topic for future research.

Reviewer_oEcZOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary
  • The paper addresses the fact that INT8 quantization of language models usually speeds up only matrix multiplication, while the other operators still move high-precision data and remain memory-bound.
  • It proposes an INT8 pipeline in which normalization, softmax and activation kernels also read and write INT8 data, with a reduction scheme that keeps on-chip memory use small.
  • The method achieves faster prompt processing and lower energy than BF16 and standard INT8 baselines on several open models and GPUs, with a small accuracy loss.
Critical strengths
  1. The motivation is clear and backed by measurements. Profiling shows that operators other than matrix multiplication take a large share of prompt-processing time, and the method is faster than BF16 and the INT8 baselines.
Critical weaknesses
  1. The comparisons and accuracy evidence are limited. The paper does not say how the INT8 baselines were run or how its kernels differ from the fused kernels already used in serving systems, which makes the novelty hard to judge. The method also has the worst perplexity of all compared methods on every model.
Questions for authors
  1. With compilation and FlashAttention, how much faster is the method than INT8 quantization of matrix multiplication alone?
2 comments

We thank the reviewer for recognizing the motivation and performance benefits of our work. We thank the reviewer for these valuable suggestions.

  1. Performance beyond INT8 matmul.

Section 5.4 (Table 6) compares our approach with SmoothQuant (INT8 matmul-only) under the same FlashAttention and torch.compile settings. Across five models, our pipeline achieves 1.03–1.07× prefill speedup and approximately 3–9% energy savings. Although these gains are modest, they demonstrate that maintaining INT8 representations across non-matmul operators remains beneficial under optimized execution. We also acknowledge that our custom CUDA kernels are not automatically fused with surrounding operators, leaving opportunities for further compiler integration.

  1. Technical distinction from existing fused kernels.

We acknowledge that existing serving frameworks, such as vLLM, already implement efficient fused RMSNorm–quantization using block-level reductions. Our contribution is not the introduction of a fundamentally new reduction algorithm, but the extension of INT8-in/INT8-out computation across multiple Transformer operators. This design is integrated across GEMM, GEMV, Softmax, and SiLU to reduce intermediate data movement. Appendix C (Figure 5) further clarifies the operator-level precision flow.

  1. Accuracy–efficiency trade-off. We acknowledge that our pipeline exhibits higher perplexity than the compared baselines, reflecting the additional quantization of non-matmul operators. To better characterize this trade-off, we have added an operator-wise precision ablation in Appendix D (Table 10). We also extended our evaluation in Appendix G to include perplexity at context lengths of 4,096 and 8,192 tokens, along with qualitative generation.

We hope these additional results and clarifications better establish the incremental contribution and practical trade-offs of extending quantization beyond matrix multiplication.

Thank you for the clear response. Yet, it seems that extending INT8 beyond matrix multiplication adds only a small speedup (1.03–1.07×), while the method has the highest perplexity among the compared methods. In my view, this trade-off does not yet make a strong enough case, so I will keep my rating.

Reviewer_7NqmOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes an INT8 inference pipeline that extends quantized tensor storage beyond matrix multiplication to supported non-matmul operators. Its central design is to read INT8 inputs, perform numerically sensitive computations in higher precision on chip, and requantize outputs before writing them to global memory. The implementation combines INT8 GEMM with fused output requantization, SIMT GEMV, and custom reduction and element-wise kernels. For the latter kernels, hierarchical warp–block reductions and recomputation reduce the need to retain full floating-point intermediate vectors. Normalization layers separating the attention and feed-forward modules are retained in BF16. Experiments across multiple LLMs, GPU architectures, and inference configurations demonstrate improvements in latency and energy efficiency while maintaining comparable model quality.

Critical strengths
  1. The paper addresses an important and practical problem in efficient LLM inference. Its focus on optimizing data movement beyond matrix multiplication provides a useful systems perspective and has potential value for improving end-to-end inference efficiency.

  2. The paper provides a comprehensive evaluation across different models, GPU architectures, and inference workloads. The experiments assess both system efficiency and model quality, including latency, energy consumption, and accuracy. The evaluation also considers optimized inference configurations, making the results more relevant.

Critical weaknesses
  1. The benefits and drawbacks of quantizing non-matmul intermediate tensors are not sufficiently demonstrated.
  • The experiments combine several changes, including low-precision matrix multiplication, output requantization, custom non-matmul kernels, and resource-management optimizations. Although the paper includes some kernel-level comparisons, it does not sufficiently isolate the contribution of each component to end-to-end performance.
  • Table 2 shows some degradation in model quality, but the experiments do not identify which quantized operators contribute to this degradation. Operator-wise sensitivity ablation studies are needed to quantify the accuracy–performance trade-off.
  1. The reported RMSNorm speedups seem questionable because both the arithmetic workload and memory traffic increase. The arithmetic-intensity analysis alone does not explain the reported speedup.
  • For RMSNorm, Appendix C.2 shows an increased arithmetic workload (W_BF16 = 4d + 1 + C_rsqrt and W_INT8 = 14d + 2 + C_rsqrt), together with increased modeled global-memory traffic (Q_BF16 = 6d bytes and Q_INT8 = 8d bytes).
  • Although arithmetic intensity increases by approximately 2.625x, this alone cannot explain the observed speedup. The authors should provide measurements that reconcile the analytical model with the reported speedup, including a consistent account of whether inputs and weights are retained on chip or reloaded in each implementation.
  1. The impact of INT8 intermediate activations on RMSNorm inputs is not sufficiently evaluated. RMSNorm is sensitive to quantization, which is why many LLM inference implementations retain higher precision in normalization and residual pathways. Although the paper introduces an INT8 RMSNorm kernel, Sections 3 and 5.1 state that some RMSNorm layers remain in BF16. However, it is unclear whether their inputs are preserved in BF16 or dequantized from INT8. Even if INT8 activations are dequantized back to BF16 before RMSNorm, the numerical information lost during quantization cannot be recovered.

  2. The precision results are questionable.

  • Sections 3 and 5.1 state that the normalization layers (typically RMSNorm) between attention and feed-forward modules are retained in BF16, whereas Section 5.4 states that RMSNorm surrounding operators uses INT8. These descriptions appear inconsistent.
  • Other precision results are also unclear, including the input/output precision at the FlashAttention interface and the precision of the residual stream.
  1. The baselines do not consider recent advances in deep kernel fusion. Although Section 5.4 includes FlashAttention and torch.compile, recent advances in kernel fusion, such as cudnn.gemm_swiglu_wrapper_sm100, FlashNormAttention (2026), and MegaKernel approaches, can avoid materializing intermediate outputs between attention and subsequent normalization operations in HBM. This significantly reduces the memory-traffic advantage of the proposed INT8 pipeline. The authors should compare against deeper kernel fusion baselines to better demonstrate the additional benefits of INT8 quantization.

  2. The hierarchical reduction strategy (thread-local partial reductions, warp-level communication, and shared-memory exchange of warp-level results) consists of well-established CUDA optimization techniques and does not appear to be novel.

Questions for authors

Please address the weaknesses listed above. In particular, please answer the following questions:

Q1. Please conduct more controlled ablation studies in which only one design factor is changed at a time, while keeping other implementation and execution settings consistent. The current experiments appear to vary multiple factors simultaneously, making it difficult to determine which components are responsible for the benefits.

Q2. The explanation for the reported RMSNorm speedup is insufficient. Please provide more detailed GPU profiling results, including cache behavior, memory traffic, and hardware resource utilization, to better explain the observed performance improvements.

Q3. Please provide a more detailed description of the numerical precision used throughout the inference pipeline, preferably with an operator-level precision plot. Specifically, please clarify the storage and computation precision of different operators and intermediate tensors.

1 comment

We thank the reviewer for these valuable suggestions. We have expanded the experimental analysis and implementation details to address these concerns.

Q1. Controlled ablation studies We agree that isolating individual operators is important for understanding their contributions. We provide an operator-wise precision ablation in Appendix D (Table 10), where we selectively retain Softmax, RMSNorm, or SiLU in BF16 while keeping the remaining quantized operators unchanged. The results show that retaining these operators in BF16 improves perplexity by at most 0.03, indicating limited additional accuracy degradation from their quantization. Additionally, Section 5.5 (Tables 7–8) compares naive and hierarchical INT8 reduction implementations under the same input/output precision, isolating the performance benefits of our reduction design.

Q2. RMSNorm GPU profiling Beyond the analysis in Section 5.5 (Tables 7–8), we have added detailed NVIDIA Nsight Compute profiling in Appendix F (Table 11) to explain the improvements of our hierarchical reduction design. These findings support our design hypothesis: reducing shared-memory pressure enables greater GPU concurrency and more effective hardware utilization, contributing to the observed RMSNorm and Softmax speedups.

Q3. Numerical precision throughout the pipeline We have added an operator-level precision-flow diagram in Appendix C (Figure 5) to clarify tensor storage and computation precision throughout the Transformer.

Our pipeline preserves INT8 tensors between supported operators. Residual connections and normalization layers separating attention and feed-forward modules remain in BF16 for numerical stability. In the FlashAttention configuration, attention inputs are dequantized to BF16, and attention outputs are requantized to INT8 for subsequent supported operators.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.