- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 3
Read review
Paper summary
The paper proposes a method for quantifying the stability of recovery from compressed or otherwise size-reduced KV records. The idea is to store a compact representation of the worst case error that could result from using the reduced size (possibly reduced to zero) version.
For fixed queries, storing the attention output of a group of rows for a specific query means the rows themselves can be discarded. For future queries, an upper bound on the error may instead be provided. Intuitively, for a softmax decision, the output may not differ even if there is significant error in the KV input.
Critical strengths
The high-level idea of quantifying KV compression or pruning error is interesting. After quantifying the error, better informed decisions can be made about what rows to evict, compress or otherwise reduce. If the error can somehow be constrained to an individual attention layer, and made sure to not propagate further through the model (which would create an indecipherable maze of small errors throughout the model), and if decisions were made on large stretches of tokens so that it does not get in the way of efficient GPU compute, then this could perhaps become a practical means of reducing both KV storage footprint and attention compute, as well as some forms of continuous attention-level compaction.
Critical weaknesses
The evaluation is surprisingly limited, testing on a small models, with fixed queries and very few examples. This part is not confidence inspiring at all. Moreover, the authors demonstrate no understanding of the costs of selective compression and quantization at an individual row level, or the cost of performing the proposed checking.
The writing is typical choppy AI prose, almost unreadable in many cases.
Questions for authors
What potential opportunities does this method create for deployments? The costs of certification, and operating at various quantization and compression levels would seem to be vastly higher than simply storing rows with higher precision and enjoying fast parallel compute.