SelfRank: Closed-Form Reranking for Low-Latency Multimodal Retrieval

Reranking with a multimodal language model orders visual document retrieval accurately but slowly. Some existing methods require relevance labels to recover accuracy after compression and ignore retriever scores. We propose SelfRank, a reranker built on a closed form rule that quantifies and restores the missing relevance information. We derive the optimal operating point under a correlation objective and establish the exact condition under which reranker scores alone are insufficient to achieve it. SelfRank further restores the reranker with a single vector on its shallow state, fitted to the model's own uncompressed score by one centered ridge solve. Across 12 datasets from ViDoRe 2 and ViDoRe 3 with two retrievers and two backbones, SelfRank comes within 1.2 pp of a full cross encoder in NDCG@5 while running up to 48x faster, pushing the Pareto frontier of accuracy against latency in visual document reranking.