FlashRank: Fast Listwise Video Reranking from Diverse Compressed Representations

Video reranking aims to improve the quality of candidate list returned by an upstream retrieval system, however, existing rerankers are limited by their latency, because of requiring re-encoding sampled frames and audio at query time. We propose FlashRank, a listwise reranker that reranks candidate vidoes based on contexutalized representations from a multimodal video index instead of raw videos. FlashRank projects cached video, audio, and text embeddings into an LLM, which jointly contextualizes the candidate list and combines modality-specific evidence for relevance estimation. We jointly adapt the query encoder with reranking objective while keeping all cached video representations fixed, enabling improvements to both first-stage retrieval and reranking without rebuilding the index. Across nine video retrieval datasets, FlashRank achieves a $74\times$ speedup over typical video-based rerankers with comparable, or even greater, nDCG@10. We further show that large scale text ranking supervision transfers effectively to video reranking, improving relevance estimation from compressed multimodal representations. Together, this makes effective video reranking practical for deployment.