acceptodds
Under review as a conference paper at ICLR 2027

QRouter: Question-Conditioned Visual Evidence Routing for Fine-Grained Visual Question Answering

Abstract

Fine-grained visual question answering (VQA) requires models to derive answers from localized but semantically decisive regions in an image, testing their ability to precisely identify and understand question-relevant visual evidence. However, recent vision-language models still encode images as dense visual token sequences, which in fine-grained VQA often contain substantial question-irrelevant information and make it difficult for the model to retrieve sparse but critical evidence for focused and precise reasoning. To address this limitation, we first provide an analytical perspective showing that the visual support produced by question-conditioned routing can concentrate answer-relevant evidence more effectively than dense question-agnostic visual prefixes. Motivated by this analysis, we present QRouter, a framework that introduces question-conditioned grounding as an intermediate structural prior for fine-grained visual question answering. QRouter uses grounded regions to reorganize dense visual features into a compact, question-relevant structured visual sequence consisting of region tokens, routed context tokens, and a background token, which respectively capture localized evidence, supporting context, and global scene information. This representation more explicitly organizes key visual evidence before multimodal reasoning, highlights the information most relevant to the question, suppresses irrelevant distractions, and provides more effective visual support for fine-grained reasoning. The resulting sequence is then processed by an efficient Mamba-based multimodal backbone. Experiments on seven benchmarks show that QRouter consistently improves over strong open-source baselines on both open-ended and grounding-sensitive question answering tasks. The improvements are particularly pronounced on benchmarks that require compositional reasoning, spatial verification, and hallucination suppression.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Multimodal Large Language Models, VQA, Question-Conditioned Grounding, Region-Aware Tokenization

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_We5bOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper studies fine-grained visual question answering, where an answer is generated from an image-question pair. QRouter first uses Qwen2.5-VL and SAM2 to perform question-conditioned grounding for each pair, then reorganizes approximately 729 dense visual tokens into 32 region tokens, 128 routed context tokens, and 1 background token, for a total of 161 visual tokens, before passing them to a Mamba-based VLM. The paper reports accuracy improvements on multiple VQA benchmarks.

Critical strengths
  1. The problem is important, and organizing visual evidence conditioned on the question is intuitive.
  2. Accuracy improves on multiple benchmarks, and the question-shuffled control also indicates that grounding uses question semantics.
Critical weaknesses

The latency numbers in Table 4 are not measured for the two systems under the same end-to-end setup. For every image-question pair, QRouter must first run Qwen2.5-VL and SAM2 to produce question-conditioned grounding; however, the main comparison in Table 4 reports only the VQA latency after grounding, while Cobra-3.5B has no corresponding extra grounding stage. Appendix Table 12 reports 0.4386 s/sample for QRouter-B's grounding stage and 0.5248 s/sample for its complete online latency, whereas the reported latency for Cobra-3.5B is 0.0857 s/sample. Thus, in the complete online inference setting, QRouter-B is approximately 6.1 times slower, and grounding accounts for about 84% of its total latency. Moreover, compressing the visual prefix from 729 tokens to 161 tokens does not yield a clear practical speedup: even excluding grounding, QRouter-B takes 0.0862 s/sample, essentially the same as Cobra's 0.0857 s/sample. The current results therefore support an accuracy-compute trade-off in which substantial extra computation is exchanged for higher accuracy, rather than the paper's claim of an efficiency advantage. Since this issue directly affects the paper's central system claim, I do not think it can be resolved by a simple clarification.

Questions for authors

Please clarify the complete measurement scope of the latencies in Table 4: does QRouter-B's 0.0862 s/sample fully exclude the Qwen2.5-VL and SAM2 grounding stages, while Cobra-3.5B's 0.0857 s/sample covers its complete online inference process? If the measurement protocols are indeed different, do the authors agree that the efficiency claim should be limited to the post-grounding VQA stage rather than interpreted as an end-to-end system efficiency advantage?

1 comment

As the discussion deadline approaches, we would appreciate your feedback on our rebuttal and any remaining questions.

Reviewer_RCpvOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes QRouter, a framework for VQA that uses question-conditioned segmentation to construct a compact visual input for a Mamba-based answering model. It combines pooled region tokens, selected context patches, and a background token, reducing the visual sequence from 729 to 161 tokens.

Experiments report improvments across seven benchmarks and two backbnone sizes, particularly on compositional and spatial reasoning tasks.

Critical strengths
  1. The method addresses a meaningful problem: locating sparce, question-relevant evidence with dense visual inputs. Combining region summaries with contextual patches and background information is a sensible design for balancing foccused evidence slection and contextual converage.
  2. The reproted improvements are substantial across several benchmarks.
  3. The analysis goes beyond just benchmark scores. Component ablations, grounding measurements, and the question-shuffled intervention provide useful evidence that grounding alignment matters to the model.
Critical weaknesses
  1. The novelty relative existing question-guided compression is insufficiently established. Prior work already conditions visual feature extraction or token selection on the question. Particularly relevant examples include InstructBLI, QG-VTC and TRIM. TRIM’s combination of question-relevant patch selection and an aggregate token for unselected patches is especially close to part of the proposed design. QRouter’s potentially distinctive contribution is the integration of explicit segmentation priors and region summaries into this process, but the paper does not directly establish the advantage of that integration over comparable alternatives.
  2. The training detail for the routing scorer is unclear. In Eqs. (6)–(7) and (32)–(35), learned scores determine hard top-LL indices, while the selected score values do not appear in the downstream token representation. Under the formulation as written, ordinary backpropagation would not provide gradients to the scoring parameters through these discrete indices.
  3. The comparisons do not adequately isolate the proposed token organization. QRouter adds a Qwen2.5-VL/SAM2 grounding system and additional segmentation supervision relative to the main baselines. Therefore, the reported gains may partly reflect the capabilities and supervision of the added grounding model. The component ablations are helpful, but comparisons using the same grounding front end, training data, and token budget with simpler pooling or selection methods would better establish the contribution of the proposed structure. Direct evaluation of the grounding VLM as an answering model would also be informative.
  4. Some experimental conclusions require stronger uncertainty reporting and methodological clarification. Main results are reported from single runs, limiting interpretation of small ablation gains. Table 8 also shows regressions on attribute and short-program questions, suggesting task-dependent benefits.
Questions for authors
  1. How will the proposed method handle multi-turn VQA? Given one image, could the proposed method adapt to follow-up questions?
  2. How are the routing scorer’s parameters trained through hard top-LL selection? Does the implementation use an estimator, auxiliary loss, or score-dependent weighting omitted from the equations?
  3. With the grounding model, supervision, and token budget held fixed, how does QRouter compare with simple mask-guided pooling/selection and existing question-guided compression methods?
  4. Which exact Qwen2.5-VL checkpoint is used, and how does it perform when answering directly? What are the complete pipeline’s parameter count, peak memory, and latency under matched evaluation settings?
Reviewer_MfajOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper addresses fine-grained VQA, where the answer depends on small, localized evidence. It argues that current VLMs feed the backbone a dense, question-agnostic prefix of visual tokens, which reduce this evidence. QRouter uses question-conditioned grounding as an intermediate structural prior instead of as an output target. A grounding module (Qwen2.5-VL with SAM2 Hiera-L) predicts soft masks with confidence scores for the question. These are projected onto the patch grid and aggregated into a patch-level relevance prior. A Region Tokenizer then builds a compact visual sequence containing region tokens, routed context tokens and a single background token. This sequence is fed to a Mamba-based backbone. The model is evaluated on seven benchmarks against open-source baselines. The paper also reports a GQA breakdown by question type, component ablations and a latency comparison.

Critical strengths
  • The motivation is clear and convincing. Many questions depend on small local details and dense, question-independent image tokens can abandon them. The paper explains this well and ties the design to it. • The results are strong and consistent. QRouter beats the compared open-source models on all seven benchmarks, at both model sizes. • The ablation study is good. Table 3 shows that adding each component (region tokens, routed context, background token) improves results, and that the two-stage training beats one-stage. • The paper includes qualitative examples (Figure 3) that show where the model fixes mistakes of the baseline.
Critical weaknesses
  • It is not fully clear where the gains come from. QRouter uses extra pretrained models (Qwen2.5-VL and SAM2) to find the relevant regions. The baselines do not have them. So it is not clear how much of the improvement comes from the proposed way of organizing tokens, and how much from the extra models being good at locating things. A comparison where a baseline receives the same region information in a simple way would make the claim more convincing. • The paper says the question-conditioned part is what helps. A control using question-independent regions (or random ones) with the same structure would show this more directly.
Questions for authors
  • How does a baseline perform if it gets the same region information in a simple way? • What happens if the regions are not question-specific (for example, question-independent or random regions) with the same token structure? • What would be the performance of just routed context, just background tokens or routed context and background tokens?
1 comment

As the discussion deadline approaches, we would appreciate your feedback on our rebuttal and any remaining questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.