- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes ByteBudget, a scheme that simultaneously optimizes weight and KV precisions for CPU LLM decoding. By (1) incorporating the effective traffic of each quantization unit into the formulation and (2) jointly selecting weight and KV cache precisions for each unit under various context lengths, the authors claim to have achieved improved latency-accuracy trade-offs.
Critical strengths
- Jointly optimizing weight and KV cache precision is well motivated.
- The problem formulation and the steps for explaining the solution are concise and clear.
- Precision allocation for quantized LLMs is an important and timely problem.
Critical weaknesses
Insufficiently comprehensive evaluation.
- As the paper's central claim is that jointly optimizing weight and KV cache precisions yields better latency-accuracy trade-offs, the evaluation should compare ByteBudget against prior approaches that optimize weight and KV cache precisions separately. The five questions currently addressed in the evaluation are mostly self-evident and, more importantly, do not directly substantiate the paper's central claim.
Insufficient motivation and CPU-specific positioning.
- The paper does not sufficiently motivate the use of CPUs for autoregressive decoding. In the Introduction, the paper states that CPU-based inference is important for private, offline, and edge-device deployments. However, this argument is only partially convincing, as GPUs and NPUs are commonly used for local/on-device LLM inference, and numerous prior works on precision allocation assume such hardware configurations [1, 2, 3]. The paper should identify specific scenarios in which CPU-based LLM decoding is necessary and explain why existing GPU-oriented precision allocation methods cannot be directly applied in these scenarios.
- The problem formulation and proposed solution are largely generic rather than CPU-specific. The same optimization formulation could also be applied to GPU-based inference. The paper lacks an in-depth analysis of the fundamental differences between CPUs and GPUs that should be considered in the problem formulation, as well as CPU-specific design choices that would distinguish the proposed approach from prior GPU-oriented precision allocation methods.
[1] Li, Shiyao, et al., "LLM-MQ: Mixed-Precision Quantization for Efficient LLM Deployment," The Efficient Natural Language and Speech Processing Workshop with NeurIPS, 2023.
[2] Kwon, Sangwoo, et al., "Dp-llm: Runtime model adaptation with dynamic layer-wise precision assignment," NeurIPS, 2025.
[3] Zhao, Junchen, et al., "CoopQ: Cooperative Game Inspired Layerwise Mixed Precision Quantization for LLMs," ACL Findings, 2026.
Questions for authors
- Please address the concerns raised in the Critical Weaknesses section.
- In the third question of the Experiments section (i.e., Figures 2(c) and (d)), the authors report the predicted total loss. How exactly is this loss computed?
- The paper states that CPU-based LLM decoding is important for private, offline, and edge-device inference. However, the experimental setup in Appendix A uses two server-grade CPU sockets with 500 GB of DRAM. As this configuration is more representative of a server environment than a local inference environment, the assumption of single-batch inference is not sufficiently justified. The authors should either evaluate ByteBudget on more representative hardware (e.g., an NVIDIA Jetson platform or a desktop-grade CPU) or provide a stronger justification for the relevance of the chosen server-grade setup and single-batch workload.
- The KV cache length increases continuously during autoregressive decoding as tokens are generated, potentially changing the optimal precision allocation over time. Does ByteBudget support runtime precision reallocation during inference? How does the system accommodate changes in the optimal weight and KV cache precisions as the context length increases? If runtime reallocation is supported, what are the associated overheads of switching precision configurations, including the potential need to re-quantize existing KV cache entries?
4 comments
Regarding Comment 1,
- While, as the authors acknowledged, the analysis and discussion of the proposed method’s impact on accuracy are indeed lacking, the core problem of the evaluation is that it does not establish a proper baseline. A comparison against GPU-based allocation methods and against a naive combination of a weight precision allocation method and a KV cache precision allocation method is necessary. Without a proper baseline, the evaluation remains incomplete.
Regarding Comment 2,
- Please provide evidence that GPU-based methods underperform on CPUs. Do HAWQ-V2 and LLM-MQ exhibit severe performance degradation when applied to CPUs, compared to when they are applied to GPUs?
Regarding Comment 3,
- I could not identify any design choices that account for the limitations of CPUs in terms of parallelism, computational throughput, and memory bandwidth. In fact, the term “CPU” does not even appear in Sections 2 and 3. Please point out the specific lines in the design section that address the differences between CPUs and GPUs.
Regarding the First Response
- Due to time constraints in the rebuttal process, it is difficult to include comprehensive verification or additional discussion; however, we intend to add comprehensive verification—particularly discussions regarding accuracy and performance—in the Camera-Ready version.
Regarding the Second Response
- We recognize that our wording was misleading. The authors do not believe that applying GPU-based methods uniformly to CPUs will necessarily result in a performance degradation.Rather, our validation of the proposed method in this paper is based on the idea that when performing CPU inference, it is necessary to consider methods suited for CPU execution. We also agree that, as you point out, a discussion of the performance gap between our method and GPU-based methods is necessary, and we recognize that additional analysis will be required for the Camera-Ready version.
Regarding the Third Response
- While we do mention CPU inference in the Abstract and Section 1, it is true, as you point out, that Sections 2 and 3 focus entirely on theoretical frameworks and mathematical explanations, with no mention of CPUs. Therefore, in the Camera-Ready version, we intend to discuss the advantages of our method for CPU inference as much as possible, while also analyzing GPU-based methods and adding the results of our comparisons and discussions.
While I appreciate the authors' efforts to address my concerns, given the lack of quantitative evidence and the significant amount of revision needed for the current version of the paper, I will maintain my current score.
Comment 1
Authors' Answer
Comment 2
Authors' Answer
Comment 3
Authors' Answer
Question 1
Authors' Answer
Question 2
Authors' Answer
Question 3
Authors' Answer
Question 4
Authors' Answer