- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper develops an INT8 LLM inference pipeline that stores intermediate tensors in INT8 across supported matmul and non-matmul operators, while performing sensitive computations in floating point on chip. It combines fused output quantization with reduction kernels that avoid full-vector storage through recomputation. Experiments on Qwen3 and Llama-3.2 report latency and energy improvements with modest quality degradation, including gains alongside FlashAttention and torch.compile.
Critical strengths
- The paper presents a coherent implementation of low-precision dataflow across supported operator boundaries. It keeps tensor storage in INT8 while retaining floating-point computation on chip, and fuses matmul output requantization to avoid higher-precision global-memory intermediates. For non-matmul operators, it accounts for the additional reductions introduced by output-scale computation, even for element-wise functions, and uses recomputation to limit temporary storage. These design choices address both global-memory traffic and on-chip resource constraints in the inference pipeline.
- The evaluation examines model quality together with latency and energy, and checks the pipeline across different models, GPUs, batch sizes, and autoregressive decoding workloads. Of particular value is the optimized configuration: both BF16 and INT8 use FlashAttention and torch.compile, and the evaluated INT8 pipeline retains latency and energy benefits. This provides evidence that the complete implementation has practical value alongside established inference optimizations, strengthening the case for the work as a systems contribution.
Critical weaknesses
- The resource-efficient kernel contribution is not sufficiently distinguished from existing implementations. Section 4.1 and Appendix A.2 clearly describe the reduction and recomputation strategy, and Tables 7–8 substantiate its performance and resource benefits against the reported baselines. However, an existing vLLM v0.8.5 RMSNorm–quantization kernel already uses the same broad three-stage organization with block-reduction temporary storage instead of a full-row buffer. Its floating-point inputs differ from the scaled INT8 inputs of Algorithm 1. The paper does not clearly distinguish which aspects of the resource strategy constitute a technical advance over this existing organization and which are adaptations to the INT8 interface. This limits the novelty assessment of a stated main contribution, while leaving the demonstrated implementation benefits intact.
- The additional end-to-end benefit beyond INT8 matmul remains insufficiently isolated in the optimized setting. Table 2 provides comparisons with existing quantization methods, Figure 4 demonstrates reduced non-matmul execution time, and Table 6 shows useful gains alongside FlashAttention and torch.compile. However, these results do not isolate the incremental gain from the extended non-matmul dataflow over a matched matmul-only INT8 pipeline under the optimized configuration. Table 6 changes both matmul precision and the surrounding dataflow relative to BF16, leaving the magnitude of this additional benefit unresolved.
I lean toward weak rejection because the technical advance over existing kernel implementations is insufficiently established, and the optimized experiments do not isolate the additional end-to-end benefit of extending quantization beyond matmul. The demonstrated practical benefits are meaningful, and clarification of these two points could change my assessment.
Questions for authors
- For RMSNorm, which aspects of Section 4 constitute new kernel techniques, and which adapt existing fused normalization–quantization designs? Please clarify the incremental contribution relative to the vLLM implementation cited above, accounting for the different input formats.
- Under the FlashAttention and torch.compile configuration of Table 6, what additional end-to-end gain comes from extending INT8 storage through the surrounding non-matmul operators beyond using INT8 matmuls? A representative paired comparison could match matmul backends and optimization settings, while allowing necessary output-format and epilogue differences.
2 comments
Thank you for the response. My score remains unchanged.
We thank the reviewer for these valuable questions and suggestions.
Q1. Incremental contribution relative to vLLM
We acknowledge that vLLM already implements an efficient fused RMSNorm–quantization kernel using a three-stage computation and block-level reductions. Our work builds on these established GPU techniques rather than introducing a fundamentally new reduction algorithm. The key distinction lies in the input/output precision and integration into the inference pipeline. The cited vLLM kernel accepts floating-point inputs and produces quantized outputs, whereas our RMSNorm kernel directly consumes INT8 inputs, performs higher-precision computation on chip, and writes INT8 outputs. Furthermore, our contribution extends beyond RMSNorm by integrating INT8 GEMM, GEMV, Softmax, and SiLU into a low-precision dataflow that preserves INT8 representations across supported operators. We will clarify this distinction in the revised manuscript.
Q2. Incremental benefits beyond INT8 matmul.
Section 5.4 (Table 6) compares SmoothQuant (INT8 matmul-only) with our full INT8 pipeline under the same FlashAttention and torch.compile optimization settings. Across five models, our pipeline provides an additional 1.03–1.07× prefill speedup and approximately 3–9% energy savings over INT8 matmul-only execution. While these gains are relatively small, they demonstrate that preserving INT8 intermediate representations remains beneficial under optimized execution. The modest improvements may partly reflect our reliance on custom CUDA kernels, which are not automatically fused with surrounding operators. We acknowledge this limitation and identify tighter integration with graph compilers and operator fusion as important directions for future work.