Evidence-Aware MaxSim Reranking for Visualized Document Retrieval

Late-interaction retrieval models such as ColPali and ColQwen2 have recently shown strong performance on the Visualized Document Retrieval (VDR) task by matching query tokens against page-level patch embeddings via MaxSim. However, we observe that MaxSim-based scoring often exhibits systematic ranking failures, where pages with peaked similarity or insufficient supporting evidence are over-ranked, leading to unreliable retrieval results. In this work, we propose EA-MaxSim, an evidence-aware reranking framework that augments MaxSim with two complementary signals: query-level evidence coverage, which measures how well a page supports the full query intent, and page-level evidence richness, which characterizes the structural sufficiency of supporting evidence within a page by leveraging spatial layout cues. EA-MaxSim operates as a lightweight reranking module, requiring no changes to model training, document encoding, or indexing. Extensive experiments on ViDoRe v2 and ViDoSeek demonstrate that EA-MaxSim consistently improves ranking quality across different backbones and datasets, particularly at top ranks. Further analyses show that evidence coverage provides a robust calibration signal, while evidence richness offers complementary improvements.