- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper introduces PAT (Predictive Abstraction Transformer), a KV-cache-efficient architecture that periodically compresses local tokens into compact representations. The paper frames the compression operation as ‘rewriting’ the given context into a more compact and informative representation. The proposed architecture showcases strong empirical performance in language modeling capabilities, consistently achieving improved perplexity compared to the variant with full self-attention despite being more efficient. PAT also showcases comparable downstream performance to full self-attention variant while using significantly less KV cache.
Critical strengths
The paper addresses an important problem of fundamentally reducing the KV cache requirements of LLMs. The empirical results are strong, with PAT consistently achieving lower perplexity than the full self-attention baseline despite using smaller KV cache, and shows comparable performance on downstream tasks with better efficiency. The paper also provides a thorough theoretical analysis to back up the approach.
Critical weaknesses
I did not find any critical weakness.
Questions for authors
I do not have any further questions that may materially affect my assessment.
Rebuttal
Thank you for your analysis of our work and the positive review!
Some of our other contributions are:
Please let us know if there is anything else we can clarify or answer for you.