- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
AquaPIC improves low-bit LLM accuracy by selecting input-dependent low-rank corrections from a bank stored in host memory or SSD. It fits the corrections using error-based partitioning and weighted low-rank approximation. Predictors use earlier activations to select and prefetch corrections for upcoming computations. Experiments on two LLMs and two quantizers cover 2-bit and 3-bit settings, with substantial gains in several 2-bit configurations and modest reported inference overhead.
Critical strengths
1.Substantial accuracy gains in aggressive quantization settings. For Llama with 2-bit QuIP# at rank 64, Table 2 reports an average downstream score of 60.4%, compared with 44.1% for the strongest listed same-rank baseline. This supports the value of specializing corrections for different inputs.
2.A concrete implementation with promising overhead. The paper implements correction selection and asynchronous transfers for host memory and SSD. Table 3 reports approximately 2.3% additional latency over a fixed resident correction. This makes the systems contribution worth considering.
Critical weaknesses
1.The accuracy and latency results are not clearly established for the same configuration. The released accuracy code defaults to correcting prompt tokens, while the fast generator defaults to leaving them uncompensated. Runtime export also requantizes the correction factors, and the supplied timing script includes prefill. The paper does not clearly map these choices to Tables 2 and 3. These code observations are confirmed, but the settings used for the reported experiments remain unclear. A matched evaluation is needed to establish that the fast runtime retains the reported accuracy.
2.The need for the full design under the stated memory constraint is insufficiently supported. Section 2 motivates the method through limited GPU memory. However, Tables 2 and 3 do not provide a comparison of quality, latency, and measured peak GPU memory under an explicit budget against feasible higher-bit models or a global correction. Figure 2 and Table 1 provide partial evidence for the partitioning and predictor, but their end-to-end advantage over simpler choices is less clear. The released prefetch path also disables correction for the first attention block; unless the non-prefetch control uses the same setting, this could confound Table 9. These gaps limit the evidence that the added complexity is justified.
Questions for authors
1.For Tables 2 and 3, which prompt-compensation policy and factor-conversion procedure were used? Please report quality from the fast runtime and separate prefill time from decoding time per output token.
2.Can one representative comparison under an explicit usable GPU-memory limit report quality, peak memory, and latency against feasible simpler alternatives? This should isolate the partitioning/predictor choice and clarify whether first-attention compensation was held constant in Table 9.
6 comments
- Memory budget is not matched for evaluation. Also, the memory usage should be reported.
Peak Device Memory. For Llama-3.1-8B, AquaPIC keeps only the partition estimators (0.078 bpw) and a reusable prefetch buffer (0.003 bpw) in device memory, 0.081 bpw in total. A 16-bit rank-64 resident compensator requires 0.384 bpw. For 4096 tokens, the KV cache requires 0.616 bpw. The table below reports the footprint, the peak memory allocated by PyTorch during decoding, and the accuracy for QuIP#.
| Configuration | Footprint | Peak memory | Avg. 0-shot |
|---|---|---|---|
| 2-bit | 2.616 bpw | 4.12 GiB | 42.4 |
| 2-bit + AquaPIC (rank 64) | 2.697 bpw | 4.19 GiB | 60.4 |
| 2-bit + rank-64 resident (best baseline) | 3.000 bpw | 4.44 GiB | 44.1 |
| 3-bit | 3.616 bpw | 5.02 GiB* | 70.9 |
| 3-bit + AquaPIC (rank 64) | 3.697 bpw | 5.09 GiB* | 71.4 |
* As the official QuIP# repository does not have a 3-bit GEMV kernel, we used the implementation at https://github.com/cnygaard/glq for peak memory measurement.
We have added a memory overhead paragraph to Section 5 of the updated manuscript.
Memory Budget and Alternatives. AquaPIC improves the accuracy-memory Pareto frontier by adding a minimal bits-per-weight (bpw) overhead on top of whichever bit-width fits the budget.
- At 2 bits, it adds only 0.081 bpw (+0.07 GiB) and raises the average 0-shot accuracy by 18.0 points (42.4 to 60.4), whereas moving from 2 to 3 bits costs 1.0 bpw (+0.90 GiB) for +28.5 points. AquaPIC thus recovers 63% of the 2-to-3-bit gap with about 8% of the memory increase. The same 0.081 bpw also improves the 3-bit model (70.9 to 71.4), so the benefit holds at whichever bit-width the budget admits.
- AquaPIC is also smaller and more accurate than resident compensation. At rank 64, it uses 4.19 vs. 4.44 GiB and reaches 60.4 vs. 44.1. At rank 32, it beats the best rank-64 resident baseline in three of the four 2-bit settings in Table 2 (within 0.8 points in the fourth) while using 0.081 vs. 0.384 bpw.
- The remaining cost is latency. Relative to the uncompensated 2-bit model, resident compensation is 2.78% slower, and AquaPIC is 5.11% slower on average (Table 3). The 3-bit model is 9.2% slower (RTX 4060, GuidedQuant), so 2-bit with AquaPIC remains faster than 3-bit.
AquaPIC’s benefit becomes more prominent as the bit-width gets lower. In our evaluated configuration (Llama-3.1-8B with QuIP#), this low-bit regime corresponds to a usable memory budget below 5.02 GiB, where 3-bit no longer fits and AquaPIC is applied to the 2-bit model. Such low-bit conditions are common because device memory is limited and the LLM must share it with other critical workloads such as the OS kernel, applications, and the display graphics driver (Section 2.2). Since these workloads take priority, industry deployments minimize the model footprint as long as accuracy remains acceptable. Apple ships its ~3B on-device model at 2 bits (~0.75 GB), even on devices with at least 8 GB of RAM [1]. NVIDIA cut the footprint of the G-Assist model, which shares the GPU with a running game, by 40% [2]. NVIDIA's Jetson guide shrinks a pipeline's VLM from 6.6 GB to 2.2 GB and recommends the lowest precision that meets the accuracy target [3].
[1] Li, Ethan, et al. “Apple Intelligence Foundation Language Models: Tech Report 2025.” arXiv 2025.
[2] NVIDIA Project G-Assist, https://www.nvidia.com/en-us/software/nvidia-app/g-assist
[3] Maximizing Memory Efficiency with Agent Skills to Run Bigger Models on NVIDIA Jetson, https://developer.nvidia.com/blog/maximizing-memory-efficiency-to-run-bigger-models-on-nvidia-jetson
- The end-to-end advantage of output-error-aligned clustering and estimator is unclear.
We evaluate the end-to-end effect of AquaPIC's components. The K-means configuration performs K-means clustering instead of the output-error-aligned clustering and utilizes the centroids as its estimator. The Centroid configuration uses the output-error-aligned clusters of AquaPIC, but replaces the MLP estimator with input centroids. All other settings follow the Ours setup of Table 2. The following table shows the WikiText-2 perplexity using 2-bit GuidedQuant.
| Llama-3.1-8B-Instruct | Qwen3-4B-Instruct-2507 | |||
|---|---|---|---|---|
| Config | Rank 32 | Rank 64 | Rank 32 | Rank 64 |
| QERA | 29.33 | 24.73 | 40.08 | 32.51 |
| Centroid | 28.44 | 24.29 | 39.32 | 34.46 |
| K-means | 28.61 | 24.05 | 38.85 | 34.21 |
| AquaPIC | 26.75 | 22.54 | 37.17 | 32.22 |
The results demonstrate the effectiveness of the design choices of AquaPIC. The Centroid configuration has limited end-to-end improvements, as the centroid-based estimator cannot select the appropriate partition. The K-means configuration, whose partitions are not aligned with the output-error objective, also underperforms AquaPIC. We have updated the manuscript and added this evaluation in Appendix D.5.
Regarding inference overhead. The centroid-based and MLP estimators have nearly identical cost: for every block, the MLP estimator has 4096×128+128×128 parameters for Llama-3.1-8B-Instruct, while the centroid-based estimator needs 4096×128 parameters, which is a 3% difference. On RTX 4060 with host memory prefetching, we measure the average decoding latency difference between the MLP estimator and the centroid-based estimator to be 0.38%.
- First attention compensation is disabled for prefetching, which confounds Table 9.
For Table 9, the first attention compensation is disabled for prefetching, while prefetching-off has every compensation enabled. This is due to the fact that the first attention block does not have a preceding computation block, and thus naturally it does not have a preceding computational window for overlapping the prefetching transfer. It also does not have a preceding residual connection to use as the estimator input. Therefore, the first attention block has the compensation path disabled.
Table 9 therefore compares the two configurations as they would actually be deployed, so its delta captures the full cost of enabling prefetching, including the omitted first-attention compensation. Since omitting a compensation path is expected only to add to the degradation, the reported delta, if anything, overstates the cost attributable to residual-based estimation alone.
Thank you for your thoughtful review.
We apologize for the confusion caused by an omission in our supplementary material. Table 3 was measured with
benchmark_decode.py, notgenerate.py. This script was unintentionally left out of the submitted code, and we have now added it.benchmark_decode.pytimes only the decoding iterations after prefill, so the reported TPOT does not include prefill time. The tokens-per-second metric ingenerate.pydoes include prefill, but it is inherited from the original gpt-fast codebase and was not used for the latency results in the paper.The runtime export code does quantize the compensators (the “correction factors” in the code), but this step was used only to build artifacts for the latency benchmark. For convenience, the latency benchmark used synthetic compensators, which the export code quantized once into the 8-bit/4-bit runtime format. No compensator that had already been quantized for the accuracy evaluation was quantized again.
The accuracy evaluation does not go through the runtime export code. The compensators in Table 2 are therefore quantized exactly as described in the paper: symmetric 8-bit RTN for rank 32 and 4-bit OPTQ for rank 64. For latency, TPOT depends on the size and format of the transferred compensators and on the kernels that apply them, not on their values. The synthetic artifacts have the same rank, bit-width, and memory layout as the evaluated compensators, so the TPOT in Table 3 applies equally to the evaluated compensators.
We agree that the released code should connect the two pipelines directly. We have revised the export code so that the compensators from the accuracy evaluation are converted into the runtime format without any additional quantization, and the additional experiment in point 3 uses this revised conversion. We have added the new export code to the supplementary code.
Compensating prompt tokens is the intended behavior of AquaPIC, and the Table 2 evaluation follows it. Every token, including prompt tokens, is compensated token-wise using residual-based partition prediction. For implementation simplicity, the fast runtime used for Table 3 implemented compensation only for the decoding phase, since Table 3 targets decoding latency. This does not change the reported TPOT, as it is measured only over decoding steps after prefill, and each decoding step performs the same computation and data transfers whether or not the prompt tokens were compensated.
Prefill Latency Overhead. The table below reports the prefill latency of 2-bit QuIP# Llama-3.1-8B-Instruct, following the evaluation setup for WikiText-2 of Table 2. We used RTX 4060 with host-memory prefetching and Jetson Orin NX with SSD prefetching. The uncompensated prefill latency is 1893.4 ms on RTX 4060 and 8128.7 ms on Jetson Orin NX. Relative to GPU-resident compensation, prompt compensation adds a prefill overhead of 2.3–3.3% on the RTX 4060 and 9.3–9.6% on the Jetson Orin NX, indicating that prompt tokens can be compensated efficiently in practice.
The paper focuses on the decoding phase, which typically dominates end-to-end latency in local inference. The prefill overhead is incurred once per prompt and is amortized as more tokens are generated. We have added these results to Appendix D.6 and will release the updated implementation after cleaning up the code.