- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
- paper proposes local attention first, with a learned head deciding whether to rerun the current step using full attention or not
- authors exploit the fact that: decoding states available after local computation in pretrained models already contain signals predictive of the benefit of global attention over local attention
- the method acheives upto 2.65x speedups while decoding due to reduced "full" attention calls
- evaluations on long-context benchmarks show competitive performance
Critical strengths
-
the paper identifies that pretrained models can predict when to use full attention or not, and uses this finding for a method to speed up decoding. the method preserves quality across tasks and models: both attention only and hybrids
-
the authors ablate that this predictability of full attention access performs better than random access to full attention -- this means that this predictability is not noise
-
the method is very practical, and can be readily integrated with inference serving infra such as vLLM
Critical weaknesses
-
my major complaint is its hard to know where the accuracy-vs-decode-time trade-off when these are presented in different tables -- right now, decoding speed ups and near-quality to full attention are demonstrated in different configurations
-
its unclear whether the method provides speedups on the reasoning tasks such as AIME (math) or LiveCodeBenchv6 (code) where lots of tokens are required to decode -- right now RULER and LongBench requires decoding on ~100 tokens.
Questions for authors
-
can the authors provide a clear figure -- where x axis is decoding time, y axis is accuracy -- to show how different configurations trade-off accuracy with decoding time
-
could the authors provide evaluation on AIME and LCBv6 where lots of tokens are required to be decoded and show a similar accuracy-decoding time trade-off there? since the authors use pretrained models, hopefully it shouldn't be hard to eval on these tasks :)