- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies mixed-precision KV-cache allocation under a fixed budget. It measures how bit-width changes across layers and KV heads affect projected attention-output error under the current configuration, uses dynamic programming to construct budget-preserving joint updates, and remeasures after each update. The final allocation is selected using model loss on a separate calibration set and frozen for perplexity and LongBench evaluation.
Critical strengths
-
A reasonable allocation criterion and clear methodology. The method considers interactions between bit-width changes and existing quantization error, with remeasurement adapting the criterion to configuration changes. The analysis also clearly distinguishes exact optimization of the local objective from full-model outcomes.
-
Consistent reported improvements and a reasonable transfer protocol. The method achieves the lowest perplexity across all 18 tested configurations. Distortion measurement and allocation selection use separate calibration sets, and allocations are frozen before LongBench evaluation, providing evidence of cross-task transfer.
Critical weaknesses
-
The methodological increment and contributions of the core components are insufficiently established. Attention-aware mixed precision and rate–distortion allocation have been studied previously, but the paper lacks controlled comparisons that remove the residual cross term, disable remeasurement, or use simpler allocation strategies at the same granularity. Since additional precision is heavily concentrated in the first layer in some configurations, a budget-matched first-layer protection baseline is also needed.
-
Limited comparison with advanced allocation methods. Experiments mainly compare Uniform, KV-sep, and an adapted KVTuner, without empirical comparisons against related allocation methods such as RDKV and AATC. KVTuner also uses a different allocation granularity and leaves some budget unused in several configurations, making it difficult to isolate the effects of the scoring criterion, search space, and realized budget.
-
Practical system benefits remain unverified. Evaluation uses eager quantization and dequantization without a packed mixed-precision cache or corresponding kernels, and nominal budgets exclude quantizer metadata. Actual memory usage, decoding latency, and calibration time are not measured, leaving deployment benefits unclear.
-
Limited quality references and generalization evidence. Experiments cover only 4B and 8B models and five LongBench tasks, with generation capped at 64 or 128 tokens. Long-chain reasoning and execution-based code generation are not evaluated. The main results also lack an unquantized FP16 reference, making the remaining quality gap difficult to assess.
Questions for authors
Please address the four concerns outlined under Critical Weaknesses.
3 comments
Thank you for the detailed responses and additional experiments. The new ablations, comparisons, and evaluations have addressed most of my main concerns. I am raising my rating from 2 to 3.
Thank You for Your Feedback and Updated Rating
Dear Reviewer M68u,
Thank you for reading our responses and for raising your rating. Your suggestions on the ablations, the baseline comparisons, the deployment measurements, and the broader evaluation made the paper more complete, and we will keep all of these additions in the final version.
Regards from the authors.
Revised PDF uploaded
Dear Reviewer M68u,
Thank you again for your detailed comments on the ablations, the baselines, deployment, and the evaluation. We have uploaded the revised PDF, with all changes in blue. The new results are in a new Section 4.6 and a new Appendix F with Tables 9–21. Your points are addressed as follows:
Regards from the authors.