acceptodds
Under review as a conference paper at ICLR 2027

AquaPIC: Boosting On-device LLM Quantization via Prefetching Input-wise Compensation

Abstract

Post-training quantization (PTQ) enables efficient local LLM inference, but aggressive quantization can substantially degrade accuracy. Recovering this lost accuracy typically requires additional correction data, but the gains are constrained by limited device memory budget. Prefetching correction data from second-tier memory can overcome this capacity limit, yet limited transfer bandwidth constrains their size, making compact low-rank compensation particularly suitable for this setting. We introduce input-partition-wise compensation (IPC), which exploits additional second-tier capacity by storing specialized low-rank compensators for different input regions without increasing the amount of data fetched per input. Building on IPC, AquaPIC combines output-error-aligned clustering, shared block-level partitions, and a lightweight residual-based partition estimator to select and prefetch compensators while the preceding block executes. Across two LLMs and two quantization methods, AquaPIC improves zero-shot accuracy under 2-bit quantization by 6.8 percentage points on average over the strongest same-rank baselines, while incurring only 2.27% average latency overhead across discrete GPU and unified memory setups.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
LLM Quantization, Post Training Quantization, Local LLM Inference, Low-rank Compensation, Efficiency, ML System

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 4 reviews, above 85% of the others.
Reviewer_8VZoOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

AquaPIC improves low-bit LLM accuracy by selecting input-dependent low-rank corrections from a bank stored in host memory or SSD. It fits the corrections using error-based partitioning and weighted low-rank approximation. Predictors use earlier activations to select and prefetch corrections for upcoming computations. Experiments on two LLMs and two quantizers cover 2-bit and 3-bit settings, with substantial gains in several 2-bit configurations and modest reported inference overhead.

Critical strengths

1.Substantial accuracy gains in aggressive quantization settings. For Llama with 2-bit QuIP# at rank 64, Table 2 reports an average downstream score of 60.4%, compared with 44.1% for the strongest listed same-rank baseline. This supports the value of specializing corrections for different inputs.

2.A concrete implementation with promising overhead. The paper implements correction selection and asynchronous transfers for host memory and SSD. Table 3 reports approximately 2.3% additional latency over a fixed resident correction. This makes the systems contribution worth considering.

Critical weaknesses

1.The accuracy and latency results are not clearly established for the same configuration. The released accuracy code defaults to correcting prompt tokens, while the fast generator defaults to leaving them uncompensated. Runtime export also requantizes the correction factors, and the supplied timing script includes prefill. The paper does not clearly map these choices to Tables 2 and 3. These code observations are confirmed, but the settings used for the reported experiments remain unclear. A matched evaluation is needed to establish that the fast runtime retains the reported accuracy.

2.The need for the full design under the stated memory constraint is insufficiently supported. Section 2 motivates the method through limited GPU memory. However, Tables 2 and 3 do not provide a comparison of quality, latency, and measured peak GPU memory under an explicit budget against feasible higher-bit models or a global correction. Figure 2 and Table 1 provide partial evidence for the partitioning and predictor, but their end-to-end advantage over simpler choices is less clear. The released prefetch path also disables correction for the first attention block; unless the non-prefetch control uses the same setting, this could confound Table 9. These gaps limit the evidence that the added complexity is justified.

Questions for authors

1.For Tables 2 and 3, which prompt-compensation policy and factor-conversion procedure were used? Please report quality from the fast runtime and separate prefill time from decoding time per output token.

2.Can one representative comparison under an explicit usable GPU-memory limit report quality, peak memory, and latency against feasible simpler alternatives? This should isolate the partitioning/predictor choice and clarify whether first-attention compensation was held constant in Table 9.

6 comments

Thank you for your thoughtful review.

  1. TPOT is measured with prefill time included.

We apologize for the confusion caused by an omission in our supplementary material. Table 3 was measured with benchmark_decode.py, not generate.py. This script was unintentionally left out of the submitted code, and we have now added it. benchmark_decode.py times only the decoding iterations after prefill, so the reported TPOT does not include prefill time. The tokens-per-second metric in generate.py does include prefill, but it is inherited from the original gpt-fast codebase and was not used for the latency results in the paper.

  1. Compensators are requantized in the runtime export code.

The runtime export code does quantize the compensators (the “correction factors” in the code), but this step was used only to build artifacts for the latency benchmark. For convenience, the latency benchmark used synthetic compensators, which the export code quantized once into the 8-bit/4-bit runtime format. No compensator that had already been quantized for the accuracy evaluation was quantized again.

The accuracy evaluation does not go through the runtime export code. The compensators in Table 2 are therefore quantized exactly as described in the paper: symmetric 8-bit RTN for rank 32 and 4-bit OPTQ for rank 64. For latency, TPOT depends on the size and format of the transferred compensators and on the kernels that apply them, not on their values. The synthetic artifacts have the same rank, bit-width, and memory layout as the evaluated compensators, so the TPOT in Table 3 applies equally to the evaluated compensators.

We agree that the released code should connect the two pipelines directly. We have revised the export code so that the compensators from the accuracy evaluation are converted into the runtime format without any additional quantization, and the additional experiment in point 3 uses this revised conversion. We have added the new export code to the supplementary code.

  1. Prompt-compensation policy and runtime quality do not match.

Compensating prompt tokens is the intended behavior of AquaPIC, and the Table 2 evaluation follows it. Every token, including prompt tokens, is compensated token-wise using residual-based partition prediction. For implementation simplicity, the fast runtime used for Table 3 implemented compensation only for the decoding phase, since Table 3 targets decoding latency. This does not change the reported TPOT, as it is measured only over decoding steps after prefill, and each decoding step performs the same computation and data transfers whether or not the prompt tokens were compensated.

Prefill Latency Overhead. The table below reports the prefill latency of 2-bit QuIP# Llama-3.1-8B-Instruct, following the evaluation setup for WikiText-2 of Table 2. We used RTX 4060 with host-memory prefetching and Jetson Orin NX with SSD prefetching. The uncompensated prefill latency is 1893.4 ms on RTX 4060 and 8128.7 ms on Jetson Orin NX. Relative to GPU-resident compensation, prompt compensation adds a prefill overhead of 2.3–3.3% on the RTX 4060 and 9.3–9.6% on the Jetson Orin NX, indicating that prompt tokens can be compensated efficiently in practice.

DeviceRankGPU ResidentOursOverhead
RTX 4060321992.5 ms2058.9 ms3.3%
642050.5 ms2097.0 ms2.3%
Jetson Orin NX328457.2 ms9247.1 ms9.3%
648692.5 ms9529.8 ms9.6%

The paper focuses on the decoding phase, which typically dominates end-to-end latency in local inference. The prefill overhead is incurred once per prompt and is amortized as more tokens are generated. We have added these results to Appendix D.6 and will release the updated implementation after cleaning up the code.

  1. Memory budget is not matched for evaluation. Also, the memory usage should be reported.

Peak Device Memory. For Llama-3.1-8B, AquaPIC keeps only the partition estimators (0.078 bpw) and a reusable prefetch buffer (0.003 bpw) in device memory, 0.081 bpw in total. A 16-bit rank-64 resident compensator requires 0.384 bpw. For 4096 tokens, the KV cache requires 0.616 bpw. The table below reports the footprint, the peak memory allocated by PyTorch during decoding, and the accuracy for QuIP#.

ConfigurationFootprintPeak memoryAvg. 0-shot
2-bit2.616 bpw4.12 GiB42.4
2-bit + AquaPIC (rank 64)2.697 bpw4.19 GiB60.4
2-bit + rank-64 resident (best baseline)3.000 bpw4.44 GiB44.1
3-bit3.616 bpw5.02 GiB*70.9
3-bit + AquaPIC (rank 64)3.697 bpw5.09 GiB*71.4

* As the official QuIP# repository does not have a 3-bit GEMV kernel, we used the implementation at https://github.com/cnygaard/glq for peak memory measurement.

We have added a memory overhead paragraph to Section 5 of the updated manuscript.

Memory Budget and Alternatives. AquaPIC improves the accuracy-memory Pareto frontier by adding a minimal bits-per-weight (bpw) overhead on top of whichever bit-width fits the budget.

  • At 2 bits, it adds only 0.081 bpw (+0.07 GiB) and raises the average 0-shot accuracy by 18.0 points (42.4 to 60.4), whereas moving from 2 to 3 bits costs 1.0 bpw (+0.90 GiB) for +28.5 points. AquaPIC thus recovers 63% of the 2-to-3-bit gap with about 8% of the memory increase. The same 0.081 bpw also improves the 3-bit model (70.9 to 71.4), so the benefit holds at whichever bit-width the budget admits.
  • AquaPIC is also smaller and more accurate than resident compensation. At rank 64, it uses 4.19 vs. 4.44 GiB and reaches 60.4 vs. 44.1. At rank 32, it beats the best rank-64 resident baseline in three of the four 2-bit settings in Table 2 (within 0.8 points in the fourth) while using 0.081 vs. 0.384 bpw.
  • The remaining cost is latency. Relative to the uncompensated 2-bit model, resident compensation is 2.78% slower, and AquaPIC is 5.11% slower on average (Table 3). The 3-bit model is 9.2% slower (RTX 4060, GuidedQuant), so 2-bit with AquaPIC remains faster than 3-bit.

AquaPIC’s benefit becomes more prominent as the bit-width gets lower. In our evaluated configuration (Llama-3.1-8B with QuIP#), this low-bit regime corresponds to a usable memory budget below 5.02 GiB, where 3-bit no longer fits and AquaPIC is applied to the 2-bit model. Such low-bit conditions are common because device memory is limited and the LLM must share it with other critical workloads such as the OS kernel, applications, and the display graphics driver (Section 2.2). Since these workloads take priority, industry deployments minimize the model footprint as long as accuracy remains acceptable. Apple ships its ~3B on-device model at 2 bits (~0.75 GB), even on devices with at least 8 GB of RAM [1]. NVIDIA cut the footprint of the G-Assist model, which shares the GPU with a running game, by 40% [2]. NVIDIA's Jetson guide shrinks a pipeline's VLM from 6.6 GB to 2.2 GB and recommends the lowest precision that meets the accuracy target [3].

[1] Li, Ethan, et al. “Apple Intelligence Foundation Language Models: Tech Report 2025.” arXiv 2025.

[2] NVIDIA Project G-Assist, https://www.nvidia.com/en-us/software/nvidia-app/g-assist

[3] Maximizing Memory Efficiency with Agent Skills to Run Bigger Models on NVIDIA Jetson, https://developer.nvidia.com/blog/maximizing-memory-efficiency-to-run-bigger-models-on-nvidia-jetson

  1. The end-to-end advantage of output-error-aligned clustering and estimator is unclear.

We evaluate the end-to-end effect of AquaPIC's components. The K-means configuration performs K-means clustering instead of the output-error-aligned clustering and utilizes the centroids as its estimator. The Centroid configuration uses the output-error-aligned clusters of AquaPIC, but replaces the MLP estimator with input centroids. All other settings follow the Ours setup of Table 2. The following table shows the WikiText-2 perplexity using 2-bit GuidedQuant.

Llama-3.1-8B-InstructQwen3-4B-Instruct-2507
ConfigRank 32Rank 64Rank 32Rank 64
QERA29.3324.7340.0832.51
Centroid28.4424.2939.3234.46
K-means28.6124.0538.8534.21
AquaPIC26.7522.5437.1732.22

The results demonstrate the effectiveness of the design choices of AquaPIC. The Centroid configuration has limited end-to-end improvements, as the centroid-based estimator cannot select the appropriate partition. The K-means configuration, whose partitions are not aligned with the output-error objective, also underperforms AquaPIC. We have updated the manuscript and added this evaluation in Appendix D.5.

Regarding inference overhead. The centroid-based and MLP estimators have nearly identical cost: for every block, the MLP estimator has 4096×128+128×128 parameters for Llama-3.1-8B-Instruct, while the centroid-based estimator needs 4096×128 parameters, which is a 3% difference. On RTX 4060 with host memory prefetching, we measure the average decoding latency difference between the MLP estimator and the centroid-based estimator to be 0.38%.

  1. First attention compensation is disabled for prefetching, which confounds Table 9.

For Table 9, the first attention compensation is disabled for prefetching, while prefetching-off has every compensation enabled. This is due to the fact that the first attention block does not have a preceding computation block, and thus naturally it does not have a preceding computational window for overlapping the prefetching transfer. It also does not have a preceding residual connection to use as the estimator input. Therefore, the first attention block has the compensation path disabled.

Table 9 therefore compares the two configurations as they would actually be deployed, so its delta captures the full cost of enabling prefetching, including the omitted first-attention compensation. Since omitting a compensation path is expected only to add to the degradation, the reported delta, if anything, overstates the cost attributable to residual-based estimation alone.

Reviewer_jRw4OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces AquaPIC, a post-training framework for improving aggressively quantized LLMs through input-partition-wise low-rank compensation. It partitions calibration activations according to quantization-induced output errors and fits specialized compensators for different input regions. During inference, lightweight MLP partition estimators predict the required compensators, which are prefetched from second-tier memory while preceding Transformer blocks execute. Experiments on two LLMs using GuidedQuant and QuIP# demonstrate improved zero-shot accuracy under aggressive quantization, with low decoding latency overhead.

Critical strengths
  1. AquaPIC combines output-error-aligned clustering, shared block-level partitions, and residual-based prefetching to exploit additional second-tier memory capacity without increasing the per-input compensator payload at a fixed rank and precision. This provides a practical approach to improving quantized LLM accuracy under limited device memory and transfer bandwidth.

  2. The paper demonstrates substantial accuracy improvements in several 2-bit quantization settings, particularly for Llama-3.1-8B with QuIP#. The implementation is evaluated on both discrete-GPU and unified-memory platforms, with only 2.27% average prefetching overhead relative to device-resident low-rank compensation during 4,096-token generation.

Critical weaknesses

1. Insufficient Component Isolation:

The downstream benefits of AquaPIC's specific clustering and partition estimation methods are not sufficiently isolated. Figure 2 compares output-error-aligned clustering against K-means and random partitioning using layer-output error, but does not establish whether these improvements translate into better downstream performance. Similarly, Table 1 reports higher partition assignment accuracy for the trained MLP estimator than for input centroid-based assignment, without comparing their end-to-end performance. Although Table 4 evaluates the estimator against oracle partition selection, comparisons with simpler alternatives are still missing. Consequently, it remains unclear how much of the reported improvement comes from the proposed components rather than input-partition-wise compensation itself.

2. Limited Comparative Accuracy–Resource Evaluation:

AquaPIC deliberately exploits additional second-tier memory capacity while keeping the per-input compensator payload bounded. Although the paper reports secondary memory requirements and evaluates different cluster counts and compensator precisions, it does not fully characterize the accuracy–resource trade-off against alternative feasible single-compensator configurations under the same device-memory and transfer-bandwidth constraints. Furthermore, the reported offline preparation costs lack directly comparable measurements for the compensation baselines, making it difficult to assess the additional computational cost of input-wise specialization.

Questions for authors
  1. Novelty and Clustering Contribution: How does AquaPIC differ technically from Quant Experts [1], which also investigates token-adaptive low-rank compensation for quantization errors in vision-language models? While Figure 2 shows improvements in layer-output error, can the authors provide end-to-end perplexity and zero-shot accuracy comparisons against K-means or random partitioning under otherwise identical settings?

  2. Partition Estimator Effectiveness: Table 1 shows that the trained MLP estimator achieves higher partition assignment accuracy than input centroid-based assignment. Does this improvement translate into meaningful downstream gains? Can the authors compare perplexity, zero-shot accuracy, and inference overhead under the same available activation information and prefetching conditions?

  3. Accuracy-Resource Trade-off: Beyond the cluster-count and compensator-precision analyses, can the authors compare AquaPIC against alternative feasible single-compensator configurations under the same device-memory and prefetch-bandwidth constraints? How do the resulting accuracy gains relate to the required secondary memory capacity, particularly in the targeted 2-bit setting?

  4. Offline Preparation Cost: Appendix B.5 reports approximately 9 hours of offline preparation for Llama-3.1-8B using eight RTX 3090 GPUs. How does this compare with QERA's compensator preparation cost under comparable hardware and calibration settings? Can the authors break down the preparation time across clustering, compensator fitting, and estimator training, and clarify how it scales with the number of clusters?


[1] Chenwei Jia, Baoting Li, Xuchong Zhang, Mingzhuo Wei, Bochen Lin, and Hongbin Sun. Quant Experts: Token-aware Adaptive Error Reconstruction with Mixture of Experts for Large Vision-Language Models Quantization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 24716-24726, 2026.

4 comments

Thank you for your thoughtful review.

  1. Novelty comparison against Quant Experts (QE) is needed.

Both QE and AquaPIC select a low-rank compensator for each token, but they differ in three key respects.

  • Partitioning.

    • QE partitions input channels, not tokens. It clusters channels by the co-occurrence of each token's top-kk important channels (∣x∣⊙meanrow∣W∣\left| x \right| \odot \mathrm{mean}_{\text{row}}\left| W \right|). It then fits each routed expert as a channel-weighted SVD of the shared expert's residual, without reference to the tokens later routed to it.
    • AquaPIC partitions tokens directly. It alternates between assigning each token to the compensator with the lowest output error and refitting each compensator in closed form from its assigned tokens' Hessian, so both steps minimize the same objective (up to Hessian damping and randomized SVD). While QE's optional refinement approaches this alignment through gradient-based training, AquaPIC builds it into the partitioning itself.
  • Prefetching suitability.

    • QE fits a router and experts for every linear layer, which conflicts with prefetching in two ways. First, routers for mid-block projections (o_proj and down_proj) need intermediate activations that exist only after the block starts, so their transfers fall on the critical path. Second, even layers sharing an input (q/k/v and gate/up) are routed independently, yielding separate, non-contiguous transfers that reduce effective link bandwidth.
    • AquaPIC instead shares partitions across all linear layers in a block. A single estimator on the pre-block residual therefore selects one contiguous set of compensators for the next block, fetched while the current block executes.
  • Non-trivial base-quantizer integration.

    • QE builds on the premise that quantization error arises mainly from a few salient input channels, which it identifies per token from ∣x∣⊙meanrow∣W∣\left| x \right| \odot \mathrm{mean}_{\text{row}}\left| W \right| using the full-precision weights. The channel groups that define its experts are therefore fixed by WW and the calibration activations alone, without reference to W^\hat{W}. QE is also not a standalone add-on to a given W^\hat{W}: its shared expert exempts the token-independent channels from quantization and applies channel-wise scaling, changing the quantized weights themselves. Applying QE to QuIP# or GuidedQuant would thus require re-running them on modified weights.
    • AquaPIC makes no assumption about channel structure and keeps the base weights frozen. It partitions tokens by their post-compensation output error using only ΔW=W−W^{\Delta}W=W-\hat{W} and calibration inputs, adapting to each quantizer's error structure, as shown with both QuIP# and GuidedQuant.

To our knowledge, no official implementation of QE has been released; the comparison above is therefore based on its paper, and we could not evaluate it under our settings.

  1. End-to-end effectiveness of output-error-aligned partitioning and estimator needs comparison against simpler alternatives, including inference overhead.

We evaluate the end-to-end effect of AquaPIC's components. The K-means configuration performs K-means clustering instead of the output-error-aligned clustering and utilizes the centroids as its estimator. The Centroid configuration uses the output-error-aligned clusters of AquaPIC, but replaces the MLP estimator with input centroids. All other settings follow the Ours setup of Table 2. The following table shows the WikiText-2 perplexity using 2-bit GuidedQuant.

Llama-3.1-8B-InstructQwen3-4B-Instruct-2507
ConfigRank 32Rank 64Rank 32Rank 64
QERA29.3324.7340.0832.51
Centroid28.4424.2939.3234.46
K-means28.6124.0538.8534.21
AquaPIC26.7522.5437.1732.22

The results demonstrate the effectiveness of the design choices of AquaPIC. The Centroid configuration has limited end-to-end improvements, as the centroid-based estimator cannot select the appropriate partition. The K-means configuration, whose partitions are not aligned with the output-error objective, also underperforms AquaPIC. We have updated the manuscript and added this evaluation in Appendix D.5.

Regarding inference overhead. The centroid-based and MLP estimators have nearly identical cost: for every block, the MLP estimator has 4096×128+128×128 parameters for Llama-3.1-8B-Instruct, while the centroid-based estimator needs 4096×128 parameters, which is a 3% difference. On RTX 4060 with host memory prefetching, we measure the average decoding latency difference between the MLP estimator and the centroid-based estimator to be 0.38%.

  1. Characterize the accuracy-resource trade-off against alternative feasible single-compensator configurations under the same device-memory and transfer constraints, and relate the gains to the required secondary memory.

Peak Device Memory. For Llama-3.1-8B, AquaPIC keeps only the partition estimators (0.078 bpw) and a reusable prefetch buffer (0.003 bpw) in device memory, 0.081 bpw in total. A 16-bit rank-64 resident compensator requires 0.384 bpw. For 4096 tokens, the KV cache requires 0.616 bpw. The table below reports the footprint, the peak memory allocated by PyTorch during decoding, and the accuracy for QuIP#.

ConfigurationFootprintPeak memoryAvg. 0-shot
2-bit2.616 bpw4.12 GiB42.4
2-bit + AquaPIC (rank 64)2.697 bpw4.19 GiB60.4
2-bit + rank-64 resident (best baseline)3.000 bpw4.44 GiB44.1
3-bit3.616 bpw5.02 GiB*70.9
3-bit + AquaPIC (rank 64)3.697 bpw5.09 GiB*71.4

* As the official QuIP# repository does not have a 3-bit GEMV kernel, we used the implementation at https://github.com/cnygaard/glq for peak memory measurement.

We have added a memory overhead paragraph to Section 5 of the updated manuscript.

Single-compensator alternatives. A single compensator in second-tier memory can be configured in two ways. If it uses the same second-tier capacity as AquaPIC, its full payload must be fetched for each block. This transfer no longer fits within the prefetching window, so it lands on the decoding critical path and the design cannot match AquaPIC’s latency, making it infeasible. If it instead obeys the same transfer constraint as AquaPIC, its rank is capped at that of AquaPIC’s per-input compensator at the same precision, because the volume transferable within the prefetching window is bounded by the link bandwidth. A single compensator therefore cannot turn additional second-tier capacity into accuracy, which is the limitation that motivates IPC. Consequently, the feasible single-compensator alternatives under the same constraints correspond to the same-rank baselines compared in Table 2. AquaPIC improves 2-bit zero-shot accuracy over the strongest of these by 6.8 percentage points on average. Keeping such a compensator resident in device memory instead yields the same accuracy as these baselines while consuming additional device memory.

Accuracy vs. secondary memory. The second-tier memory consumption is directly proportional to the number of clusters. Table 10 in Appendix D.4 shows the effect of scaling the number of clusters, which demonstrates the general trend of quality improvement as the number of clusters increases. Thus, the cluster count is a knob that the user can adjust for a preferable accuracy-secondary memory usage trade-off.

  1. The offline preparation cost needs more detail.

Baseline Offline Cost. We measured single-compensator QERA on the same workstation (eight RTX 3090 GPUs) with the same calibration set. QERA takes about 1.1 hours for Llama-3.1-8B-Instruct, while AquaPIC with 128 clusters takes about 9 hours, which is a one-time offline cost. Note that the official QERA implementation does not use multiple GPUs efficiently and took more than 20 hours in our setup, so we used our own multi-GPU implementation of QERA.

The runtimes in Appendix B.5 are end-to-end and include calibration activation forwarding, the output-error-based clustering iterations with per-cluster compensator fitting, and partition estimator training. Calibration activation forwarding takes about 0.6 hours, about 8.2 hours for the clustering iterations, and 0.2 hours for estimator training. Because tokens are partitioned rather than duplicated across clusters, Hessian accumulation costs the same as for a single compensator. The per-cluster overhead is mainly the Cholesky factorization and SVD, while randomized SVD and early exiting keep this cost modest.

We have added the baseline offline cost and time breakdown in Appendix B.5.

I appreciate the authors' comprehensive and detailed response, which address most of my previous concerns. I would keep my initial score of 3. Thank you

Thank you for taking the time and effort to review our paper and consider our response. Your constructive feedback has helped us improve the manuscript.

Reviewer_YEDSOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces AquaPIC for recovering accuracy lost through aggressive quantization in local LLM inference under limited device memory. The method constructs specialized low-rank compensators for different input partitions, stores them in host memory or SSD storage, and uses a lightweight partition estimator with residual-based early prediction to prefetch the required compensators while the preceding block executes. Experiments cover two LLMs, two base quantization methods, and 2-bit and 3-bit settings, demonstrating accuracy improvements over single-compensator methods with modest additional decoding latency on the evaluated devices.

Critical strengths

The problem is clearly defined and well motivated. The paper distinguishes device-memory capacity constraints from transfer-bandwidth constraints and motivates increasing the number of compensators rather than their individual rank. Input specialization, shared block-level partitions, and early prefetching form a coherent design, and the presentation clearly connects these components to the target deployment setting.

Critical weaknesses

1. Missing baseline comparison of offline preparation cost AquaPIC introduces clustering, repeated compensator fitting, and partition-estimator training beyond single-compensator approaches. Appendix B.5 reports approximately 9 hours for Llama-3.1-8B-Instruct and 7.5 hours for Qwen3-4B-Instruct-2507 on a workstation with eight RTX 3090 GPUs, but does not report the preparation cost of single-compensator baselines under comparable conditions. Consequently, the additional computational cost required to obtain the accuracy gains from input specialization remains unclear, limiting assessment of the overall cost-benefit trade-off.

2. Missing direct comparison of peak runtime device-memory usage Limited device-memory capacity is a central motivation of the paper. Although Appendix C describes compensator-buffer reuse and Appendix B.4 reports secondary-memory usage, the evaluation does not directly compare total peak runtime device-memory usage across AquaPIC, device-resident compensation, and the uncompensated quantized model. The actual device-memory savings, including the additional memory required by the estimators and buffers, are therefore not sufficiently quantified.

3. Limited analysis of the cost of routing errors Table 1 reports top-1 partition-assignment accuracies ranging from 66.9% to 82.5% for the evaluated layers, indicating a non-negligible rate of incorrect partition selection. Although Table 4 reports the aggregate perplexity impact of using the estimator, it does not show the distribution of compensation errors associated with incorrect routing. The current results therefore do not establish whether routing errors are generally inexpensive or whether a small subset of inputs incurs substantial additional error that is obscured by aggregate metrics.

Questions for authors
  1. How much additional offline computation does AquaPIC require compared with single-compensator QERA, and which preparation stages are included in the runtimes reported in Appendix B.5?

  2. What is the peak runtime device-memory usage of AquaPIC compared with device-resident compensation and the uncompensated quantized model? How much device memory does AquaPIC actually save?

  3. When the estimator selects a non-optimal partition, what is the distribution of the additional compensation error relative to the optimal partition? Are there infrequent but high-cost routing errors, and how does this change when using residual-based early prediction?

2 comments

Thank you for your thoughtful review.

  1. Offline cost comparison against baseline is missing.

Baseline Offline Cost. We measured single-compensator QERA on the same workstation (eight RTX 3090 GPUs) with the same calibration set. QERA takes about 1.1 hours for Llama-3.1-8B-Instruct, while AquaPIC with 128 clusters takes about 9 hours, which is a one-time offline cost. Note that the official QERA implementation does not use multiple GPUs efficiently and took more than 20 hours in our setup, so we used our own multi-GPU implementation of QERA.

The runtimes in Appendix B.5 are end-to-end and include calibration activation forwarding, the output-error-based clustering iterations with per-cluster compensator fitting, and partition estimator training. Calibration activation forwarding takes about 0.6 hours, about 8.2 hours for the clustering iterations, and 0.2 hours for estimator training. Because tokens are partitioned rather than duplicated across clusters, Hessian accumulation costs the same as for a single compensator. The per-cluster overhead is mainly the Cholesky factorization and SVD, while randomized SVD and early exiting keep this cost modest.

We have added the baseline offline cost and time breakdown in Appendix B.5.

  1. The memory usage should be reported.

Peak Device Memory. For Llama-3.1-8B, AquaPIC keeps only the partition estimators (0.078 bpw) and a reusable prefetch buffer (0.003 bpw) in device memory, 0.081 bpw in total. A 16-bit rank-64 resident compensator requires 0.384 bpw. For 4096 tokens, the KV cache requires 0.616 bpw. The table below reports the footprint, the peak memory allocated by PyTorch during decoding, and the accuracy for QuIP#.

ConfigurationFootprintPeak memoryAvg. 0-shot
2-bit2.616 bpw4.12 GiB42.4
2-bit + AquaPIC (rank 64)2.697 bpw4.19 GiB60.4
2-bit + rank-64 resident (best baseline)3.000 bpw4.44 GiB44.1
3-bit3.616 bpw5.02 GiB*70.9
3-bit + AquaPIC (rank 64)3.697 bpw5.09 GiB*71.4

* As the official QuIP# repository does not have a 3-bit GEMV kernel, we used the implementation at https://github.com/cnygaard/glq for peak memory measurement.

We have added a memory overhead paragraph to Section 5 of the updated manuscript.

  1. Analysis of routing errors is needed.

We analyze routing errors on 2-bit GuidedQuant Llama-3.1-8B-Instruct with rank-64 compensators, with clustering on RedPajama and evaluation on WikiText-2. We use the q_proj and up_proj of Layer 16 listed in Table 1. For each token xx, we compute the output error reduction of each of the 128 partitions, ∥Wx−W^x∥2−∥Wx−(W^+BcAc)x∥2\|Wx - \hat{W}x\|_2 - \|Wx - (\hat{W} + B_cA_c)x\|_2, i.e., the output error norm of the quantized layer minus that with the compensator of partition cc, and rank the partitions from best to worst. We then average the error reduction of the i-th best partition over all tokens (i = 1, …, 128) and compare these rank-wise means with the mean error reduction of the partition selected by the estimator. We evaluate two settings, one using the block's actual input (Estimator) and one using the pre-block residual as in prefetching (Residual estimator).

Figure A(a) (https://anonymous.4open.science/r/ICLR27_Submission3851/FigureA_routing_error_analysis.pdf) plots the rank-wise means as a histogram, with the two estimator means as vertical lines. Three observations follow.

  • Selecting the best partition is important, as even the second-best partition achieves only 58-79% of the best partition's mean error reduction.
  • Both estimators stay close to the best partition. The Estimator retains 96.1-97.6% of the best partition's mean error reduction, and the Residual estimator retains 95.5-96.1%.
  • All 128 rank-wise means are positive for both projections. Even the worst-ranked partition reduces the output error on average, so a routing error typically lowers the gain rather than adding error. Together, these observations show that routing errors cost little on average, even when the estimator predicts from the pre-block residual.

Since averages alone cannot rule out rare high-cost errors, Figure A(b) shows the token-wise distribution of the error reduction gap to the best partition, i.e., the error reduction of the best partition minus that of the selected partition, with the 90th and 99th percentiles marked. More than 60% of the tokens fall in the first bin, which contains all correctly routed tokens, whose share corresponds to the top-1 accuracy in Table 1. Beyond it, the token share decays monotonically, with no secondary mode at large gaps. High-cost routing errors are therefore rare. With residual-based early prediction, the up_proj distribution almost overlaps with that of the Estimator. The q_proj distribution shifts moderately toward larger gaps, yet the Residual estimator still retains 96.1% of the best partition's mean error reduction for q_proj. This is consistent with the small perplexity increase from prefetching in the 2-bit GuidedQuant rows of Table 9 (0.10-0.39).

We have added this analysis to Appendix E of the revised manuscript.

Reviewer_H6uTOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

AquaPIC improves aggressively quantized on-device LLMs through input-adaptive low-rank error compensation. Instead of increasing compensation rank, it stores multiple specialized compensators in secondary memory and fetches only the selected set, increasing correction capacity without increasing the per-input payload at a fixed rank and precision. Its main contributions are output-error-aligned clustering, shared partitions across each Attention or FFN block, and a lightweight residual-based estimator that enables compensator prefetching during the preceding block’s execution. Base model weights remain frozen; compensators and estimators are fitted using calibration data.

Evaluation covers Llama-3.1-8B and Qwen3-4B with GuidedQuant and QuIP# at 2 and 3 bits. Under 2-bit quantization, AquaPIC improves average zero-shot accuracy by 6.8 percentage points over the strongest same-rank compensation baselines. Llama-8B decoding measurements on RTX 4060 and Jetson Orin NX show average latency overheads of 2.27% relative to resident compensation and 5.11% relative to uncompensated quantization. Ablations indicate that approximate routing, early prediction, and compensator quantization retain most of the accuracy gains.

Critical strengths
  1. A compelling algorithm–system co-design. AquaPIC increases compensation capacity through input-specific low-rank corrections while keeping the selected payload fixed at a given rank and precision. Output-error-aligned clustering, block-level partitions, and early residual-based prediction make this idea compatible with prefetching. This directly addresses the interaction between limited device memory and limited transfer bandwidth, giving the method a clear practical motivation.
  2. Substantial accuracy improvements supported by complementary experiments. Across two models and two quantization methods, AquaPIC achieves an average 6.8 percentage-point zero-shot accuracy improvement over the strongest same-rank compensation baselines under 2-bit quantization. Ablations further show that practical routing, prefetch prediction, and compensator quantization preserve most of the gains. This supports the effectiveness of the deployable method beyond idealized compensator selection.
  3. Concrete implementation with low measured decoding overhead. The paper demonstrates both host-memory prefetching on RTX 4060 and SSD-backed execution on Jetson Orin NX, integrated with CUDA Graphs. Average overhead is 2.27% relative to resident compensation and 5.11% relative to uncompensated quantization. These results materially strengthen the deployment case by showing that accessing specialized corrections from secondary storage can remain inexpensive on the tested platforms.
Critical weaknesses
  1. The deployment advantage is not established under matched resource budgets. Same-rank comparisons demonstrate the benefit of specialized compensation, but the paper lacks a complete comparison of accuracy, peak device memory, and latency against higher-bit quantization or larger resident compensators. For example, Llama’s 2-bit QuIP# with rank-64 AquaPIC achieves 60.4% average zero-shot accuracy, while uncompensated 3-bit QuIP# achieves 70.9%. Without explicit memory constraints and matched latency measurements, it remains unclear when AquaPIC provides the preferable deployment tradeoff. This leaves its central practical advantage insufficiently demonstrated.

  2. The low-overhead claim needs stronger measurement evidence. Runtime evaluation covers Llama-8B under 2-bit quantization with a 4,096-token decoding workload, but does not report repeated-run variability, tail latency, or a transfer–computation timeline. Because successful prefetching depends on available execution windows and storage behavior, average TPOT alone cannot establish how consistently transfers are hidden. This limits confidence in the robustness of the reported 2.27% incremental overhead, which is central to the systems contribution.

Questions for authors
  1. What usable device-memory budget motivates the evaluated configurations, and what is their peak memory consumption, including estimators, compensation buffers, and KV cache? Under that budget, are 3-bit quantization or higher-rank resident compensators feasible alternatives, and does existing evidence establish AquaPIC’s accuracy–latency advantage?

  2. Do you have existing traces or component timings substantiate that compensator transfers are largely overlapped with computation?

4 comments

Thank you for your thoughtful review.

  1. Memory budget is not matched for evaluation. Also, the memory usage should be reported.

Peak Device Memory. For Llama-3.1-8B, AquaPIC keeps only the partition estimators (0.078 bpw) and a reusable prefetch buffer (0.003 bpw) in device memory, 0.081 bpw in total. A 16-bit rank-64 resident compensator requires 0.384 bpw. For 4096 tokens, the KV cache requires 0.616 bpw. The table below reports the footprint, the peak memory allocated by PyTorch during decoding, and the accuracy for QuIP#.

ConfigurationFootprintPeak memoryAvg. 0-shot
2-bit2.616 bpw4.12 GiB42.4
2-bit + AquaPIC (rank 64)2.697 bpw4.19 GiB60.4
2-bit + rank-64 resident (best baseline)3.000 bpw4.44 GiB44.1
3-bit3.616 bpw5.02 GiB*70.9
3-bit + AquaPIC (rank 64)3.697 bpw5.09 GiB*71.4

* As the official QuIP# repository does not have a 3-bit GEMV kernel, we used the implementation at https://github.com/cnygaard/glq for peak memory measurement.

We have added a memory overhead paragraph to Section 5 of the updated manuscript.

Memory Budget and Alternatives. AquaPIC improves the accuracy-memory Pareto frontier by adding a minimal bits-per-weight (bpw) overhead on top of whichever bit-width fits the budget.

  • At 2 bits, it adds only 0.081 bpw (+0.07 GiB) and raises the average 0-shot accuracy by 18.0 points (42.4 to 60.4), whereas moving from 2 to 3 bits costs 1.0 bpw (+0.90 GiB) for +28.5 points. AquaPIC thus recovers 63% of the 2-to-3-bit gap with about 8% of the memory increase. The same 0.081 bpw also improves the 3-bit model (70.9 to 71.4), so the benefit holds at whichever bit-width the budget admits.
  • AquaPIC is also smaller and more accurate than resident compensation. At rank 64, it uses 4.19 vs. 4.44 GiB and reaches 60.4 vs. 44.1. At rank 32, it beats the best rank-64 resident baseline in three of the four 2-bit settings in Table 2 (within 0.8 points in the fourth) while using 0.081 vs. 0.384 bpw.
  • The remaining cost is latency. Relative to the uncompensated 2-bit model, resident compensation is 2.78% slower, and AquaPIC is 5.11% slower on average (Table 3). The 3-bit model is 9.2% slower (RTX 4060, GuidedQuant), so 2-bit with AquaPIC remains faster than 3-bit.

AquaPIC’s benefit becomes more prominent as the bit-width gets lower. In our evaluated configuration (Llama-3.1-8B with QuIP#), this low-bit regime corresponds to a usable memory budget below 5.02 GiB, where 3-bit no longer fits and AquaPIC is applied to the 2-bit model. Such low-bit conditions are common because device memory is limited and the LLM must share it with other critical workloads such as the OS kernel, applications, and the display graphics driver (Section 2.2). Since these workloads take priority, industry deployments minimize the model footprint as long as accuracy remains acceptable. Apple ships its ~3B on-device model at 2 bits (~0.75 GB), even on devices with at least 8 GB of RAM [1]. NVIDIA cut the footprint of the G-Assist model, which shares the GPU with a running game, by 40% [2]. NVIDIA's Jetson guide shrinks a pipeline's VLM from 6.6 GB to 2.2 GB and recommends the lowest precision that meets the accuracy target [3].

[1] Li, Ethan, et al. “Apple Intelligence Foundation Language Models: Tech Report 2025.” arXiv 2025.

[2] NVIDIA Project G-Assist, https://www.nvidia.com/en-us/software/nvidia-app/g-assist

[3] Maximizing Memory Efficiency with Agent Skills to Run Bigger Models on NVIDIA Jetson, https://developer.nvidia.com/blog/maximizing-memory-efficiency-to-run-bigger-models-on-nvidia-jetson

  1. Additional evidence for successful overlapping is needed.

Statistics. The TPOT in Table 3 is averaged over three 4,096-token decoding runs, about 12k steps in total. We additionally measured the latency of each step. The table below shows the per-step distribution pooled over the three runs, which covers both step-to-step and run-to-run variation. Unlike Table 3, where TPOT is computed by averaging the total decoding time over all 4,096 steps in each run, the statistics here are based on individually timed steps and therefore differ slightly.

RTX 4060Jetson Orin NX
Rank32643264
median38.630 ms38.989 ms155.232 ms158.425 ms
p9038.645 ms38.998 ms155.342 ms158.505 ms
p9538.648 ms39.005 ms155.422 ms158.628 ms
p9938.721 ms39.129 ms155.477 ms158.698 ms

Averaged over the two ranks, p90, p95, and p99 exceed the median by only 0.012 ms, 0.017 ms, and 0.116 ms on RTX 4060, and by 0.095 ms, 0.197 ms, and 0.259 ms on Jetson Orin NX, respectively. Even at p99, the gaps remain below 0.30% on RTX 4060 and 0.17% on Jetson Orin NX. This shows that the low overhead holds consistently across steps and runs, including on the SSD-backed setup.

Trace. We provide an annotated snapshot (https://anonymous.4open.science/r/ICLR27_Submission3851/trace_overlapping_512_annotated.png) of a 512-token decoding trace captured with Nsight Systems on RTX 4060. The trace shows that in AquaPIC, the prefetching kernel overlaps with the computation of the preceding block, which keeps the latency overhead low. We cannot share the raw Nsight Systems report, as it contains identifying information that would break anonymity.

Thank you for the comprehensive response, which addresses most of my concerns. I will keep my original score of 3.

Thank you for the comprehensive response, which addresses most of my concerns. I will keep my original score of 3.

Comments on the paper

1 comment
Area_Chair_YRvfOpenReview ↗

Action Required: Respond to Author Rebuttals Before the Discussion Period Ends

Dear Reviewers,

The authors have posted their rebuttals. As the author–reviewer discussion period is coming to a close soon, please read their responses and participate in the discussion.

All reviewers are expected to respond, at minimum with a brief acknowledgment confirming that the reviewers have read and considered the authors’ rebuttals. If your concerns have been addressed or remain unresolved, please briefly indicate this in your response.

Thank you for your time and effort.

Best, Your AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.