RegionMax-WA: Supervising MaxSim Winners for Visual Document Retrieval

Visual document retrieval preserves the layout and visual context needed to search complex pages, but its usual supervision identifies only which page is relevant, not which evidence should support the match. This gap matters for late-interaction retrievers, whose page score sums a separate patch decision for each query token. We introduce RegionMax-WA, which supervises evidence membership in the same full-page token-wise competition used at retrieval time. OCR text and boxes propose query-dependent candidate evidence; Winner Allocation encourages each content token’s MaxSim winner to fall within it; and the auxiliary loss supplements standard page-ranking training. Across four ViDoRe V2 collections, RegionMax-WA improves macro nDCG@5 (0–100 scale) by 2.21 and 4.21 absolute points over matched page-only PaliGemma-3B and Qwen2.5-VL-3B baselines. It also exceeds a full-complement aggregate-loss control by 0.49 and 0.95 points. On 250 independently annotated query–page pairs, MaxSim winners show greater spatial agreement with human evidence boxes. OCR is used only during training; inference retains the same index format and MaxSim scorer.