ReToken: One Token to Improve Vision–Language Models for Visual Retrieval

Long visual context poses a challenge for vision-language models: performance degrades as the number of distractors grows, and processing all tokens at once is computationally infeasible under GPU memory constraints. We present ReToken, a single learnable embedding trained as an explicit retrieval target that selects a sparse set of query-relevant visual tokens from a pre-filled visual KV cache. Trained on only a small image-QA dataset, ReToken yields consistent gains across image and video benchmarks: on Visual Haystacks it improves Qwen3VL-8B by 12 points (>20% relative), and on LVBench it transfers zero-shot to long video for an 8.5-point gain. Thanks to its lightweight design, both training and long-video inference fit on a single H100. We will release our code.