QRouter: Question-Conditioned Visual Evidence Routing for Fine-Grained Visual Question Answering

Fine-grained visual question answering (VQA) requires models to derive answers from localized but semantically decisive regions in an image, testing their ability to precisely identify and understand question-relevant visual evidence. However, recent vision-language models still encode images as dense visual token sequences, which in fine-grained VQA often contain substantial question-irrelevant information and make it difficult for the model to retrieve sparse but critical evidence for focused and precise reasoning. To address this limitation, we first provide an analytical perspective showing that the visual support produced by question-conditioned routing can concentrate answer-relevant evidence more effectively than dense question-agnostic visual prefixes. Motivated by this analysis, we present QRouter, a framework that introduces question-conditioned grounding as an intermediate structural prior for fine-grained visual question answering. QRouter uses grounded regions to reorganize dense visual features into a compact, question-relevant structured visual sequence consisting of region tokens, routed context tokens, and a background token, which respectively capture localized evidence, supporting context, and global scene information. This representation more explicitly organizes key visual evidence before multimodal reasoning, highlights the information most relevant to the question, suppresses irrelevant distractions, and provides more effective visual support for fine-grained reasoning. The resulting sequence is then processed by an efficient Mamba-based multimodal backbone. Experiments on seven benchmarks show that QRouter consistently improves over strong open-source baselines on both open-ended and grounding-sensitive question answering tasks. The improvements are particularly pronounced on benchmarks that require compositional reasoning, spatial verification, and hallucination suppression.