acceptodds
Withdrawn from ICLR 2027

ByteBudget: Optimizing Weight–KV precision allocation for CPU inference

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Autoregressive decoding on CPUs is memory-bandwidth-bound; generating each token requires streaming model weights, whose traffic grows with the context length. However, the quantization methods typically optimize weights or KV caches separately, and a fixed precision policy cannot account for changes in context length, hardware characteristics, or kernel efficiency. We introduce ByteBudget, a hardware-and-context-aware planner that converts a decode-latency target into a calibrated effective-traffic budget and jointly allocates precision across weight blocks and a KV cache to minimize estimated quality loss. The framework consists of three components: a format-aware CPU cost model, modular quality-loss curves supplied by off-the-shelf quantizers, and a discrete budgeted optimizer. For practical deployment, we use an efficient marginal-gain planner to produce mixed-precision weight and KV cache configurations, with a dynamic-programming oracle and lagrangian lower bound to audit solution quality. Since KV cache traffic grows with context length while weight traffic remains roughly constant, ByteBudget adapts precision allocation changing deployment conditions. Our framework integrates the quantization methods with CPU-based LLM decoding kernels and allocates precision across layers to satisfy deployment-specific latency constraints, rather than a uniform bit-width.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM Quantization, Weight-KV Precision Allocation, CPU Infrerence for LLM, CPU-Based LLM Decoding, Bit-width optimization

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_6GdJOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ByteBudget, a scheme that simultaneously optimizes weight and KV precisions for CPU LLM decoding. By (1) incorporating the effective traffic of each quantization unit into the formulation and (2) jointly selecting weight and KV cache precisions for each unit under various context lengths, the authors claim to have achieved improved latency-accuracy trade-offs.

Critical strengths
  • Jointly optimizing weight and KV cache precision is well motivated.
  • The problem formulation and the steps for explaining the solution are concise and clear.
  • Precision allocation for quantized LLMs is an important and timely problem.
Critical weaknesses

Insufficiently comprehensive evaluation.

  • As the paper's central claim is that jointly optimizing weight and KV cache precisions yields better latency-accuracy trade-offs, the evaluation should compare ByteBudget against prior approaches that optimize weight and KV cache precisions separately. The five questions currently addressed in the evaluation are mostly self-evident and, more importantly, do not directly substantiate the paper's central claim.

Insufficient motivation and CPU-specific positioning.

  • The paper does not sufficiently motivate the use of CPUs for autoregressive decoding. In the Introduction, the paper states that CPU-based inference is important for private, offline, and edge-device deployments. However, this argument is only partially convincing, as GPUs and NPUs are commonly used for local/on-device LLM inference, and numerous prior works on precision allocation assume such hardware configurations [1, 2, 3]. The paper should identify specific scenarios in which CPU-based LLM decoding is necessary and explain why existing GPU-oriented precision allocation methods cannot be directly applied in these scenarios.
  • The problem formulation and proposed solution are largely generic rather than CPU-specific. The same optimization formulation could also be applied to GPU-based inference. The paper lacks an in-depth analysis of the fundamental differences between CPUs and GPUs that should be considered in the problem formulation, as well as CPU-specific design choices that would distinguish the proposed approach from prior GPU-oriented precision allocation methods.

[1] Li, Shiyao, et al., "LLM-MQ: Mixed-Precision Quantization for Efficient LLM Deployment," The Efficient Natural Language and Speech Processing Workshop with NeurIPS, 2023.

[2] Kwon, Sangwoo, et al., "Dp-llm: Runtime model adaptation with dynamic layer-wise precision assignment," NeurIPS, 2025.

[3] Zhao, Junchen, et al., "CoopQ: Cooperative Game Inspired Layerwise Mixed Precision Quantization for LLMs," ACL Findings, 2026.

Questions for authors
  • Please address the concerns raised in the Critical Weaknesses section.
  • In the third question of the Experiments section (i.e., Figures 2(c) and (d)), the authors report the predicted total loss. How exactly is this loss computed?
  • The paper states that CPU-based LLM decoding is important for private, offline, and edge-device inference. However, the experimental setup in Appendix A uses two server-grade CPU sockets with 500 GB of DRAM. As this configuration is more representative of a server environment than a local inference environment, the assumption of single-batch inference is not sufficiently justified. The authors should either evaluate ByteBudget on more representative hardware (e.g., an NVIDIA Jetson platform or a desktop-grade CPU) or provide a stronger justification for the relevance of the chosen server-grade setup and single-batch workload.
  • The KV cache length increases continuously during autoregressive decoding as tokens are generated, potentially changing the optimal precision allocation over time. Does ByteBudget support runtime precision reallocation during inference? How does the system accommodate changes in the optimal weight and KV cache precisions as the context length increases? If runtime reallocation is supported, what are the associated overheads of switching precision configurations, including the potential need to re-quantize existing KV cache entries?
4 comments

Comment 1

Authors' Answer

  • As noted in the comments, we acknowledge that the analysis and discussion of the impact of the proposed method on accuracy in this paper are insufficient. Therefore, we will add an analysis and discussion of the experimental results regarding accuracy. Due to time constraints, the “Rebuttals” version is limited to a discussion of two models; however, in the “Camera-Ready” version, we will conduct a comprehensive analysis of all models used for validation in this paper.

Comment 2

Authors' Answer

  • First, because CPUs and GPUs have different hardware characteristics—such as the types of computations they excel at and their memory bandwidth—the authors believe that applying techniques effective for GPUs directly to CPU inference will yield only limited results.Furthermore, as various companies have announced ultra-multicore CPUs designed for servers, we believe that establishing LLM inference techniques using CPUs will be beneficial for LLM inference utilizing these processors. We believe that by effectively utilizing CPUs for LLM inference—an area that has traditionally been considered the domain of GPUs—it is possible to achieve low-power, high-performance LLM inference.

Comment 3

Authors' Answer

  • The proposed method was designed with a focus on the disadvantages that CPUs face compared to GPUs in terms of the number of parallel processes, computational performance, and memory bandwidth. The proposed method can also be applied to GPUs. However, given the characteristics of GPUs and CPUs, we believe it is particularly effective for inference on CPUs.

Question 1

  • Please address the concerns raised in the Critical Weaknesses section.

Authors' Answer

  • We will address the issues you have pointed out and make the necessary revisions.

Question 2

  • In the third question of the Experiments section (i.e., Figures 2(c) and (d)), the authors report the predicted total loss. How exactly is this loss computed?

Authors' Answer

  • This chart shows the results obtained by actually using an LLM model to vary the context length and bytes per token.

Question 3

  • The paper states that CPU-based LLM decoding is important for private, offline, and edge-device inference. However, the experimental setup in Appendix A uses two server-grade CPU sockets with 500 GB of DRAM. As this configuration is more representative of a server environment than a local inference environment, the assumption of single-batch inference is not sufficiently justified. The authors should either evaluate ByteBudget on more representative hardware (e.g., an NVIDIA Jetson platform or a desktop-grade CPU) or provide a stronger justification for the relevance of the chosen server-grade setup and single-batch workload.

Authors' Answer

  • We believe that the experimental environment used in this study is ideal for verifying the effectiveness of our method. While this method is likely to be effective in edge environments such as Jetson, our primary goal in this paper is to demonstrate that CPU-based inference is effective on local servers and similar systems; we do not intend to discuss its effectiveness in edge environments.

Question 4

  • The KV cache length increases continuously during autoregressive decoding as tokens are generated, potentially changing the optimal precision allocation over time. Does ByteBudget support runtime precision reallocation during inference? How does the system accommodate changes in the optimal weight and KV cache precisions as the context length increases? If runtime reallocation is supported, what are the associated overheads of switching precision configurations, including the potential need to re-quantize existing KV cache entries?

Authors' Answer

  • We do not currently support reassignments that are already in progress, and we consider this a task for the future.

Regarding Comment 1,

  • While, as the authors acknowledged, the analysis and discussion of the proposed method’s impact on accuracy are indeed lacking, the core problem of the evaluation is that it does not establish a proper baseline. A comparison against GPU-based allocation methods and against a naive combination of a weight precision allocation method and a KV cache precision allocation method is necessary. Without a proper baseline, the evaluation remains incomplete.

Regarding Comment 2,

  • Please provide evidence that GPU-based methods underperform on CPUs. Do HAWQ-V2 and LLM-MQ exhibit severe performance degradation when applied to CPUs, compared to when they are applied to GPUs?

Regarding Comment 3,

  • I could not identify any design choices that account for the limitations of CPUs in terms of parallelism, computational throughput, and memory bandwidth. In fact, the term “CPU” does not even appear in Sections 2 and 3. Please point out the specific lines in the design section that address the differences between CPUs and GPUs.

Regarding the First Response

  • Due to time constraints in the rebuttal process, it is difficult to include comprehensive verification or additional discussion; however, we intend to add comprehensive verification—particularly discussions regarding accuracy and performance—in the Camera-Ready version.

Regarding the Second Response

  • We recognize that our wording was misleading. The authors do not believe that applying GPU-based methods uniformly to CPUs will necessarily result in a performance degradation.Rather, our validation of the proposed method in this paper is based on the idea that when performing CPU inference, it is necessary to consider methods suited for CPU execution. We also agree that, as you point out, a discussion of the performance gap between our method and GPU-based methods is necessary, and we recognize that additional analysis will be required for the Camera-Ready version.

Regarding the Third Response

  • While we do mention CPU inference in the Abstract and Section 1, it is true, as you point out, that Sections 2 and 3 focus entirely on theoretical frameworks and mathematical explanations, with no mention of CPUs. Therefore, in the Camera-Ready version, we intend to discuss the advantages of our method for CPU inference as much as possible, while also analyzing GPU-based methods and adding the results of our comparisons and discussions.

While I appreciate the authors' efforts to address my concerns, given the lack of quantitative evidence and the significant amount of revision needed for the current version of the paper, I will maintain my current score.

Reviewer_kmZtOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

ByteBudget jointly allocates weight and KV-cache precision for batch-one CPU decoding under a latency budget. It combines a context-dependent traffic model, additive quality-loss estimates, and a greedy allocator, with dynamic programming and Lagrangian bounds for evaluating surrogate solution quality.

Critical strengths
  1. Joint weight–KV allocation under context-dependent CPU latency constraints addresses a relevant deployment problem.

  2. The framework explicitly considers format-dependent kernel efficiency and appropriately limits its optimality certificates to the additive surrogate.

Critical weaknesses
  1. The main performance claim is not yet supported by end-to-end measurements. Figure 2b reports predicted throughput, while Figures 2c–d primarily present predicted surrogate loss. These results do not demonstrate improved downstream quality at matched measured CPU latency. Figure 2d also ambiguously compares reportedly measured baseline results with a simulated ByteBudget frontier.

  2. Inadequate baselines. Uniform weight quantization with FP16 KV does not isolate the benefit of joint allocation. Comparisons against uniform W/K/V combinations, weight-only mixed precision, KV-only tuning, and independently tuned weight/KV configurations are missing.

  3. Unvalidated modeling assumptions and limited novelty assessment. The additive quality surrogate, format-aware latency predictor, and greedy optimality gap lack quantitative validation. I would also like a clearer comparison with WKVQuant, KV Pareto, and HAWQ-V3 to understand what is new in the allocation method.

Questions for authors
  1. Could the authors materialize ByteBudget configurations and report actual perplexity, long-context task accuracy, and CPU decode latency against the strongest supported W/K/V baselines?

  2. How accurately do the quality surrogate and latency model predict held-out joint configurations? Please report ranking correlations, latency prediction errors, and budget violation rates.

  3. What are the measured greedy–DP and certified gaps, and do better surrogate solutions consistently improve downstream quality?

2 comments

Comment 1

  • The main performance claim is not yet supported by end-to-end measurements. Figure 2b reports predicted throughput, while Figures 2c–d primarily present predicted surrogate loss. These results do not demonstrate improved downstream quality at matched measured CPU latency. Figure 2d also ambiguously compares reportedly measured baseline results with a simulated ByteBudget frontier.

Authors' Answer

  • As you pointed out, we recognized that the discussion regarding the impact of applying the proposed method on the accuracy of the LLM model was insufficient. Therefore, we will add an analysis and discussion of the experimental results regarding accuracy and revise the content to reflect the end-to-end discussion.

Comment 2

  • Inadequate baselines. Uniform weight quantization with FP16 KV does not isolate the benefit of joint allocation. Comparisons against uniform W/K/V combinations, weight-only mixed precision, KV-only tuning, and independently tuned weight/KV configurations are missing.

Authors' Answer

  • In the revision for the rebuttals, we will add a discussion on changes in performance and accuracy based on the results obtained using a mix of precision levels—both before and after applying ByteBudget. This is because this is the area where we most want to highlight the strengths of our method. However, as noted, it is true that we have not yet provided a comprehensive discussion of accuracy; therefore, we will add a comprehensive verification and analysis of accuracy for the camera-ready version.

Comment 3

  • Unvalidated modeling assumptions and limited novelty assessment. The additive quality surrogate, format-aware latency predictor, and greedy optimality gap lack quantitative validation. I would also like a clearer comparison with WKVQuant, KV Pareto, and HAWQ-V3 to understand what is new in the allocation method.

Authors' Answer

  • The novelty of the method presented in this paper lies in the fact that, with a focus on CPU inference for LLMs, it explores the limits of what can be achieved while minimizing accuracy and performance degradation by combining various optimization techniques, and identifies resource allocations that actually contribute to performance improvements.

Question 1

  • Could the authors materialize ByteBudget configurations and report actual perplexity, long-context task accuracy, and CPU decode latency against the strongest supported W/K/V baselines?

Authors' Answer

  • We will add an analysis and discussion of the experimental results regarding accuracy and performance. While the “Rebuttals” version discusses the accuracy of two models, the “Camera-Ready” version will comprehensively measure, analyze, and discuss the accuracy of all models used in the measurements in this paper.

Question 2

  • How accurately do the quality surrogate and latency model predict held-out joint configurations? Please report ranking correlations, latency prediction errors, and budget violation rates.

Authors' Answer

  • As mentioned above, we have also included an analysis of accuracy, and we will add a more comprehensive analysis in the camera-ready version.

Question 3

  • What are the measured greedy–DP and certified gaps, and do better surrogate solutions consistently improve downstream quality?

Authors' Answer

  • We recognize that, due to a lack of discussion regarding accuracy, it is difficult to see the effectiveness of the proposed method. Therefore, we will revise the content to make its effectiveness clearer.

Thank you for your responses and for acknowledging my concerns.

However, the responses mainly describe planned revisions rather than providing concrete experimental results. In particular, I could not find quantitative evidence addressing the key concerns regarding end-to-end performance, the accuracy of the quality and latency models, and the optimality gap of the proposed allocation method.

While I appreciate the authors' commitment to addressing these issues, the current evidence is insufficient to resolve my main concerns. Therefore, I will maintain my original rating.

Reviewer_XBjrOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper attempts to address the problem of how to allocate quantization precision across different model layers and the KV cache to preserve model output quality as much as possible, given specific hardware constraints, context lengths, and inference latency requirements. It proposes a framework called ByteBudget, which models the relationship between hardware characteristics (primarily memory bandwidth) and inference latency, as well as the impact of model weight and KV cache precision on output quality. Based on these models, the paper formulates the problem as an optimization task that aims to minimize model quality degradation through precision allocation under a limited bandwidth budget (i.e., a constraint on data transfer volume). It then employs a greedy search algorithm that progressively increases precision from the lowest available level to find an optimal solution. The problem addressed in this paper has practical significance. However, the proposed method mainly combines existing standard optimization techniques, offering limited novelty. The experimental evaluation is insufficient, lacking critical end-to-end experiments and validation of the key algorithmic components. Furthermore, the overall presentation of the paper is poor.

Critical strengths

Jointly optimizing the quantization precision of model weights and the KV cache is a reasonable and practically relevant research problem, as both are essential components of LLM inference. Moreover, the KV cache grows with increasing context length, making it necessary to carefully allocate resources between model weights and the KV cache in resource-constrained systems to improve inference efficiency while preserving model output quality.

Critical weaknesses
  1. In terms of methodology, this paper mainly relies on established optimization techniques, such as the Multiple-Choice Knapsack Problem, Pareto Pruning, and Dynamic Programming, offering limited novelty. Moreover, one of the most important aspects of the proposed method is modeling the relationship between the quantization of model weights and the KV cache and model quality. However, the paper approximates the overall model quality loss as a weighted sum of the quantization losses of individual layer weights and the KV cache, without experimentally verifying whether this formulation can accurately represent the actual model error. For model weight quantization, the paper uses weight reconstruction error to represent the quantization loss, but does not experimentally validate whether this is reasonable. For the KV cache, the paper estimates quality loss using calibration data, but does not specify the source of the calibration data or the evaluation metrics used. Since the proposed method is built upon this quality loss model, validating its effectiveness is necessary.
  2. In the experimental section, the paper lacks critical experiments, and the existing experiments are poorly presented and insufficiently analyzed. For example, it lacks key end-to-end experiments comparing weight-only quantization, KV-only quantization, independent optimization of weight and KV cache quantization, and joint optimization using ByteBudget. It also lacks validation of the model quality loss model, as mentioned in the previous point. In the existing experimental results, the titles of the plots in Figure 2 fail to clearly convey their main content.
  3. There are numerous issues with the presentation of the paper. The paper is excessively formalized. Although the research problem is relatively straightforward, the paper uses a large number of equations. For example, Equations (6)–(15) could be omitted or expressed through simple textual descriptions. In Section 3, subsection titles such as "3.1 COST CALIBRATION," "3.2 CURVE CONSTRUCTION AND PARETO PRUNING," and "3.3," among others, do not adequately represent the content of their respective sections or explain their logical relationships with the surrounding sections, resulting in poor readability.
  4. The inconsistent or inaccurate use of technical terminology makes the paper difficult to read. For example, terms such as Model Size, Memory Traffic, and Memory Bandwidth are used interchangeably, and the presentation of Figure 2(a) is confusing.
Questions for authors

The proposed method is built upon a quality loss model and a hardware-aware performance estimation model. Have the authors experimentally validated the effectiveness of these two models?

2 comments

Comment 1

  • In terms of methodology, this paper mainly relies on established optimization techniques, such as the Multiple-Choice Knapsack Problem, Pareto Pruning, and Dynamic Programming, offering limited novelty. Moreover, one of the most important aspects of the proposed method is modeling the relationship between the quantization of model weights and the KV cache and model quality. However, the paper approximates the overall model quality loss as a weighted sum of the quantization losses of individual layer weights and the KV cache, without experimentally verifying whether this formulation can accurately represent the actual model error. For model weight quantization, the paper uses weight reconstruction error to represent the quantization loss, but does not experimentally validate whether this is reasonable. For the KV cache, the paper estimates quality loss using calibration data, but does not specify the source of the calibration data or the evaluation metrics used. Since the proposed method is built upon this quality loss model, validating its effectiveness is necessary.

Authors' Answer

  • This paper clearly differs from previously proposed approaches in that it combines and evaluates various optimization techniques to identify the conditions under which LLM inference can realistically be performed on a CPU while minimizing accuracy degradation within limited resources. Since the criticism that our accuracy verification was insufficient is valid, we have added an evaluation of accuracy improvements.Due to time constraints, the revisions for the rebuttal will be based on evaluations of only two models; however, in the camera-ready version, we will verify and discuss the effectiveness of the proposed method for all models evaluated in this paper.

Comment 2

  • In the experimental section, the paper lacks critical experiments, and the existing experiments are poorly presented and insufficiently analyzed. For example, it lacks key end-to-end experiments comparing weight-only quantization, KV-only quantization, independent optimization of weight and KV cache quantization, and joint optimization using ByteBudget. It also lacks validation of the model quality loss model, as mentioned in the previous point. In the existing experimental results, the titles of the plots in Figure 2 fail to clearly convey their main content.

Authors' Answer

  • As noted in the first comment, it is true that there is a lack of analysis and discussion of the experimental results regarding the degree of accuracy improvement achieved by applying the proposed method. Therefore, we will add a section analyzing and discussing the experimental results related to accuracy improvement. Regarding the title of Figure 2, we will revise it to make it clearer and better describe the figure.

Comment 3

  • There are numerous issues with the presentation of the paper. The paper is excessively formalized. Although the research problem is relatively straightforward, the paper uses a large number of equations. For example, Equations (6)–(15) could be omitted or expressed through simple textual descriptions. In Section 3, subsection titles such as "3.1 COST CALIBRATION," "3.2 CURVE CONSTRUCTION AND PARETO PRUNING," and "3.3," among others, do not adequately represent the content of their respective sections or explain their logical relationships with the surrounding sections, resulting in poor readability.

Authors' Answer

  • First, regarding the comment that the mathematical expressions are overly formalized, we will revise the main text to focus only on the equations necessary for understanding the proposed method, while moving detailed discussions and proofs to the Appendix to improve the readability of the main text.

Comment 4

  • The inconsistent or inaccurate use of technical terminology makes the paper difficult to read. For example, terms such as Model Size, Memory Traffic, and Memory Bandwidth are used interchangeably, and the presentation of Figure 2(a) is confusing.

Authors' Answer

  • We will review the terminology and revise it to ensure it does not hinder understanding.

Questions

  • The proposed method is built upon a quality loss model and a hardware-aware performance estimation model. Have the authors experimentally validated the effectiveness of these two models?

Authors' Answer

  • We will add the missing analysis and discussion you pointed out. We believe that by including a discussion on accuracy, we will be able to explain the significance of the effectiveness of the proposed method in this paper.

Thanks for the response. It would be nice to add the necessary experiments and explanations, as you mentioned, for the future submission.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.