ReState: Enabling Cross-Query KV Reuse for Training-Free Acceleration of BERT Rerankers

Rerankers play a key role in retrieval-augmented generation (RAG) by refining initially retrieved rankings with models specifically trained to score query–document relevance, including commonly used BERT-based cross-encoders. However, rerankers may repeatedly encode unchanged documents across queries, making reuse of precomputed document representations a natural way to reduce redundant computation. Yet document states in BERT rerankers depend on both query interactions and query-dependent positions, preventing exact reuse across queries. But can't encoders reuse keys and values? Reranking targets relevance ordering, motivating us to rethink the need to recompute every intermediate document state. To reduce repeated encoding, we propose ReState, a training-free method for approximate reuse of document keys and values (K/V) in BERT rerankers. Our central idea is to separate full-document reading from document-state updating: query and score tokens remain active and attend to every document position, while only selected document states are updated under the current query. To reduce quality loss from query-dependent document positions, we propose Taylor-based position approximation to compensate for positional changes through precomputed correction components. To recover ranking quality with limited recomputation, we design Score-anchored restoration to select update locations using offline score-token attention, without query-time model computation. Across six benchmarks with a frozen MiniLM reranker and prepared caches, ReState uses default routing and 20% document updates to achieve a $6.857\times$ mean online pipeline speedup while retaining 96.0% of Full's mean nDCG@10.